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CHAPTER  1 


Introduction 


So  the  principles  which  are  set  forth 
in  this  treatise  will,  when  taken  up  by 
thoughtful  minds,  lead  to  many 
another  more  remarkable  result;  and 
it  is  to  be  believed  that  it  will  be  so 
on  account  of  the  nobility  of  the 
subject,  which  is  superior  to  any 
other  in  nature. 

— Galileo  Galilei  (1 638) 

Command  and  Control 
Assessment  Challenges 

NATO  and  its  member  nations  are  in  the  midst  of  a 
revolution  in  military  affairs.  There  are  three  major 
dimensions  to  this  revolution — a  geopolitical 
dimension,  a  technological  dimension,  and  a  closely 
coupled  conceptual  dimension.  This  multidimensional 
revolution  poses  significant  new  challenges  for 
analysis  in  general  and  for  command  and  control 
assessment  in  particular. 

The  changed  geopolitical  context  is  characterised  by 
a  shift  from  a  preoccupation  with  a  war  involving  NATO 
and  the  Warsaw  Pact  to  a  concern  for  a  broad  range 
of  smaller  military  conflicts  and  Operations  Other  Than 
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War  (OOTW).  Analysts  will  increasingly  be  called  upon 
to  provide  insights  into  these  non-traditional  operations. 

Advances  in  technology,  particularly  information- 
related  technologies,  offer  military  organisations 
unprecedented  opportunities  to  significantly  reduce  the 
fog  and  friction  traditionally  associated  with  conflict. 
At  the  same  time,  they  may  prove  to  be  challenges  in 
themselves  across  a  wide  variety  of  realms — technical, 
organisational,  and  cultural. 

To  the  extent  that  they  can  be  achieved,  significantly 
reduced  levels  of  fog  and  friction  offer  an  opportunity 
for  the  military  to  develop  new  concepts  of  operations, 
new  organisational  forms,  and  new  approaches  to 
Command  and  Control  (C2),  as  well  as  to  the 
processes  that  support  it.  Analysts  will  be  increasingly 
called  upon  to  work  in  this  new  conceptual  dimension 
in  order  to  examine  the  impact  of  new  information- 
related  capabilities  coupled  with  new  ways  of 
organising  and  operating. 

Definition  of  Command  and  Control 

C2  has  been  defined  by  NATO  as  Military  Function 
01:  “The  Organisation,  Process,  Procedures  and 
Systems  necessary  to  allow  timely  political  and  military 
decisionmaking  and  to  enable  military  commanders 
to  direct  and  control  military  forces.”  (NATO  1 996)  C2 
systems  are  further  defined  in  NATO  documents  to 
include:  headquarters  facilities,  communications, 
information  systems,  and  sensors  &  warning 
installations.  (NATO  1998). 

Other  terms  are  used  in  NATO  member  nations  that 
are  synonymous  with,  or  closely  related  to,  C2.  These 
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include  Command,  Control,  and  Communications  or 
Consultation  (C3),  and  Computers  (C4),  and 
Intelligence  ([C3I]  or  [C4I]),  and  Surveillance  and 
Reconnaissance  (C4ISR).  The  term  CIS  is  sometimes 
used  to  refer  to  command  information  systems.  More 
recently  the  term  “C2”  has  referred  to  the  collaborative 
and  consultative  processes  that  are  an  inherent  part 
of  coalition  operations. 

For  the  purposes  of  this  Code  of  Best  Practice  (COBP), 
the  term  C2  is  intended  to  be  an  umbrella  term  that 
encompasses  the  concepts,  issues,  organisations, 
activities,  processes,  and  systems  associated  with  the 
NATO  definition  of  C2  as  well  as  the  other  terms 
enumerated  above. 

Uniqueness  of  C2  Analyses  and  Issues 

The  focus  of  military  research  and  analysis  has 
predominantly  been  on  the  physical  domain.  C2  issues 
differ  in  fundamental  ways  from  physics  dominated 
problems.  C2  deals  with  distributed  teams  of  humans 
operating  under  stress  and  in  a  variety  of  other  operating 
conditions.  C2  problems  are  thus  dominated  by  their 
information,  behavioural,  and  cognitive  aspects  that 
have  been  less  well  researched  and  understood.  This 
focus  creates  a  multidimensional,  complex  analytic 
space. 

Military  operations  involve  multi-sided  dynamics 
encompassing  friendly,  adversary,  and  other  actors 
including: 

•  Action-reaction  dynamics; 
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•  Tightly  connected  interaction  among  subjective 
elements  such  as  cultures,  morale,  doctrine, 
training,  and  experience  and  between  those 
subjective  elements  and  the  combat  arena; 

•  Non  Governmental  Organisations  (NGO); 

•  Private  Volunteer  Organisations  (PVO); 

•  International  organisations; 

•  International  corporations;  and 

•  Transnational,  subnational,  criminal,  and 
terrorist  organisations. 

C2  issues  are  difficult  to  decompose  and  recompose 
without  committing  errors  of  logic.  Moreover,  the 
composition  rules,  by  which  the  various  factors  that 
impact  C2  interact,  are  poorly  understood  except  in 
arenas  that  have  been  previously  studied  in  detail. 
Finally,  the  C2  arena  is  weakly  bounded  by  issues 
that  on  initial  examination  appear  quite  finite,  but  prove 
to  be  linked  to  very  high-level  factors.  For  example, 
tactical  performance  may  be  tied  to  national  culture. 

Analyses  of  C2  are  also  often  constrained  by  factors 
that  are  beyond  the  boundaries  of  the  research.  For 
example,  security  policies  may  restrict  data  availability 
and  otherwise  constrain  the  analysis.  The  availability 
of  data  often  limits  the  scope  of  an  analysis.  Moreover, 
the  time  and  resources  available  to  conduct  an  analysis 
are  often  severely  constrained  because  the  decision 
processes  being  supported  are  being  driven  by  outside 
planning,  operational,  or  budget  and  decision 
processes.  This  should  be  seen  as  a  challenge  rather 
than  a  problem.  Uncertainty  and  risk  associated  with  a 
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lack  of  appropriate  data  need  to  be  embraced  as  part 
of  the  analytical  approach.  It  is  unreasonable  to  expect 
that  data  would  be  available  for  the  performance  of 
future  systems  and  processes  that  do  not  yet  exist.  An 
experimental  component  and  a  modelling  and 
simulation  component  need  to  be  integrated  into  modern 
C2  analyses  in  order  to  close  the  gap  in  knowledge 
and  data. 

Finally,  because  of  the  complexity  of  C2  processes 
and  systems,  analysis  in  this  area  requires  the  ability 
to  understand  how  Dimensional  Parameters  (DP), 
Measures  of  Performance  (MoP),  Measures  of  C2 
Effectiveness  (MoCE),  Measures  of  Force 
Effectiveness  (MoFE),  and  Measures  of  Policy 
Effectiveness  (MoPE)  are  linked  and  impact  on  one 
another.  The  cumulative  set  of  these  measures  is 
denoted  as  Measures  of  Merit  (MoM)  in  the  COBP. 
Determining  the  precise  nature  of  these  relationships 
nearly  always  proves  to  be  an  analytic  challenge. 

Taken  together,  all  these  factors  mean  that  C2  modelling 
and  analysis  are  more  uncertain  and  therefore  more 
prone  to  risk  than  their  equivalents  in  conventional 
weapon  and  platform  analyses.  Indeed,  C2  issues  have 
long  been  regarded  as  difficult  to  analyse.  Many 
operational  analysis  (OA)  studies  have  simply  assumed 
perfect  C2  in  order  to  focus  on  other  variables.  As  a 
result  of  these  characteristics  of  C2  analysis,  these 
endeavours  will  require  a  heavy  element  of  research 
within  each  analysis.  This  COBP  is  intended  to  assist 
the  community  in  dealing  with,  and  overcoming,  the 
barriers  to  effective  analysis  of  C2. 
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Differences  Across  the  Mission  Spectrum 

There  are  significant  differences  among  the  different 
parts  of  the  mission  spectrum  (e.g.,  MoM)  that  the 
assessment  team  needs  to  take  into  consideration. 
Table  1 .1  highlights  the  differences  between  traditional 
combat  and  OOTW. 


“ — - — - 1  - 

Mtoloivoptftlkin _ _ _ 

Stability  Relatively  stable 

May  be  more  dynamic 

Focei _ Enseny _ 

Commitment  Common  (military} 

No  traditional  opponent _ 

Uncertain  (politlcal/milltaryl 

Principles 

Unity  Of  command 

Of  purpose 

Decisionmaking  Hierarchical 

Consensus 

Operations  Surprise,  secrecy 

Transparency 

Information 

Nature  of  the  problem  Known  unknowns 

Unknown  unknowns 

Key  question  How  to  get  informabon 

Focus  Enemy  military 

Wtiat  information  to  get 
Mllitary.'po41bcalieconom*c.i 
social  factors 

Situation  awareness  Common  air-land-sea 

Limited  dissemination, 
more  complex 

Database*  Very  large,  well  structured 

Largei ,  less  structured 

Analysis 

Unit  Battalion  level  entity 

Em  In  integration  1  Nlittiiff  my 

Focus  Military  (systems,  organisations! 

Approach  [  Traditional  operation  analysis 

More  behavioural 

Very  difficult 

Poldcali'Military  and  socaetal 
“Softer”  analysis 

Table  1 .1 .  Comparison  of  Symmetric,  Conventional  Warfare  and 
OOTW  Missions  &  Principles  Mission 

In  symmetric  conventional  warfare,  the  mission  tends 
to  be  relatively  stable,  there  is  a  clear  focus  on  the 
enemy,  and  the  military  has  a  common  understanding 
and  commitment.1  Conversely,  in  OOTW  the  mission 
is  often  more  dynamic.  This  is  captured  by  the  often 
pejorative  term  “mission  creep.”  In  many  of  the 
operations  in  question  there  is  no  “enemy.”  This  is 
obviously  true  for  operations  such  as  humanitarian 
assistance  and  disaster  relief.  In  addition, 
peacekeeping  activities  involve  protagonists  who  must 
be  treated  even-handedly  if  the  operation  is  to  be 
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successful.  In  the  latter  missions,  political-military 
ambiguities  frequently  result  in  uncertain 
understanding  of  the  goals  and  objectives  of  the 
mission  and  a  limited  commitment.2  (Starr,  Haut  & 
Hughes,  1997) 

Principles 

Military  theorists  have  frequently  propounded  basic 
principles  of  conventional  warfare.  Three  often  cited 
principles  include  the  need  for  unity  of  command,  the 
importance  of  hierarchical  decisionmaking,  and  the 
criticality  of  achieving  surprise  in  operations.  A  recent 
book  has  proposed  alternative  principles  for  OOTW. 
(Alberts  &  Hayes,  1995)  It  cites  the  need  for  unity  of 
purpose,  consensus  decisionmaking,  and 
transparency  of  operations. 

Information 

In  conventional  warfare,  the  issue  of  information 
gathering  and  management  focuses  on  the  issue  of 
“known  unknowns”  (e.g.  Where  are  the  enemy’s 
battalions?).  For  that  case,  the  key  question  is  how  to 
get  the  needed  information  (e.g.  What  are  the  key 
signatures  for  the  targets  in  question?  What  sensors 
should  we  task  to  exploit  those  signatures?).  Clearly, 
the  focus  is  on  the  enemy  military  and  one  objective  is 
to  assemble  a  complete,  timely,  and  accurate  common 
picture  of  the  air-land-sea  situation.  The  result  is  a 
very  large,  time-sensitive  database,  but  one  that  is 
relatively  well  structured  (e.g.  enemy  order  of  battle). 
Conversely,  in  OOTW,  the  problem  of  information 
gathering  and  management  is  dominated  by  “unknown 
unknowns.”  Thus,  the  primary  question  to  address  is 
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what  information  to  get.  The  information  focus  is  much 
more  diffuse  because  of  the  myriad  of  military,  political, 
economic,  and  social  factors  that  must  be  considered. 
Consequently,  situation  awareness  is  much  more 
complex.  Political  considerations  often  make  it  prudent 
to  limit  the  dissemination  of  information,  creating  a 
tension  between  the  desire  to  create  shared 
awareness  by  increasing  information  sharing  and  the 
need,  for  political  and/or  security  reasons,  to  limit 
information  sharing.  The  resulting  databases  are 
frequently  larger  and  less  structured. 

Analysis 

Over  many  years,  the  military  operations  research 
community  has  become  relatively  adept  at  analysing 
key  aspects  of  symmetric  conventional  warfare.  As  an 
illustration,  analyses  of  ground  warfare  often  focus  on 
battalion-level  operations  and  techniques  have  emerged 
to  integrate  across  those  results  to  derive  insights  into 
campaign  outcomes.  The  focus  is  on  military  systems 
and  organisations,  and  the  techniques  in  question 
involve  a  broad  set  of  methods  (e.g.,  mathematical 
programming,  decision  theoretic  approaches)  and  tools 
(e.g.,  models  and  simulations).  Analyses  of  C2  issues 
remain  among  the  most  challenging,  even  in  warfare 
contexts.  In  addition,  analyses  of  OOTW  often  require 
consideration  of  individual  behaviour.  It  has  proven  very 
difficult  to  integrate  across  these  results  to  derive  a 
comprehensive  understanding  of  the  problem.  The  issue 
is  compounded  by  the  many  factors  that  have  to  be 
considered  in  the  analysis  process  (e.g.  military,  political, 
economic,  social).  This  has  led  to  the  application  of 
“softer”  analytic  approaches  (e.g.,  extensive  reliance 
on  expert  elicitation).  Moreover,  the  very  nature  of 
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warfare  appears  to  be  changing.  For  example, 
asymmetric  threats  are  becoming  more  common, 
information  technologies  are  impacting  C2  processes, 
and  organic  structure  and  dynamics  are  changing 
rapidly  and  in  ways  we  do  not  fully  understand. 

Types  of  C2  Assessments 

The  assessment  team  could  be  called  upon  to  support 
a  wide  variety  of  sponsors  (e.g.  acquirers  of  C2 
systems,  long  range  planners  and  programmers, 
developers  of  requirements,  operational  commanders, 
and  trainers).  These  sponsors  will  bring  different 
problems  to  the  assessment  team  (e.g.  assessment 
of  alternative  systems  or  concepts,  identification  and 
selection  of  alternative  courses  of  action  in  an 
operational  context).  Some  of  these  will  deal  with  a 
specific  mission  (e.g.  air  defence)  while  others  will  need 
to  deal  with  the  entire  mission  spectrum  from  forward 
presence  to  high  intensity  conventional  war. 

Specific  problems  that  the  team  may  be  called  upon 
to  address: 

Requirements  Analysis 

•  Derivation  of  specific  C2  requirements  from 
broad  statements  of  objectives;  and 

•  The  establishment  of  a  minimum,  standard,  or 
expected  level  of  performance. 
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Assessment  of  Alternatives 

•  Comparison  and  selection  of  alternative  systems 
that  may  be  very  dissimilar  but  are  designed  to 
achieve  a  similar  purpose; 

•  Assessment  of  the  utilisation  of  a  system  in  a  new 
or  unexpected  application  domain  or  mission; 

•  Trade-offs  between  C2  systems  and  combat  systems; 

•  Analysis  of  the  impact  of  an  organisational  change; 

•  Determination  of  the  most  cost-effective 
approach  to  achieving  the  desired  objective;  and 

•  Comparison  of  a  replacement  system  or 
components  of  a  system. 

Research  Issues 

•  Effectiveness  of  human  decisionmaking  as  a 
function  of  system  performance  or  other  factors; 

•  Effectiveness  of  C2  training;  and 

•  Impact  of  collaboration  on  C2  quality. 

Support  to  Operations 

•  Course  of  action  analysis; 

•  Real  time  assessment  of  mission  effectiveness;  and 

•  Rehearsal  assessment. 
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Purpose  and  Scope  of  the  COBP 

This  COBP  offers  broad  guidance  on  the  assessment 
of  C2  for  the  purposes  of  supporting  a  wide  variety  of 
decisionmakers  and  the  conduct  of  C2  research 
described  above.  It  should  be  noted  that  this  COBP  is 
focused  upon  the  assessment  challenges  associated 
with  the  nature  of  C2  and  does  not  attempt  to 
specifically  address  the  unique  properties  and 
constraints  associated  with  each  of  the  many  C2- 
related  problem  domains. 

Given  the  increasing  interdependence  among  the 
elements  of  a  mission  capability  package3 
(organisation;  doctrine;  C2  concepts,  processes, 
systems;  materiel;  education;  training;  and  forces),  C2- 
related  analysis  cannot  easily  be  done  in  isolation  from 
a  more  comprehensive  mission  analysis.  This  COBP 
is  meant  to  support  analyses  that  go  beyond  the 
traditional  boundaries  of  C2  analyses. 

This  new  version  of  the  COBP  for  C2  assessment  was 
developed  by  SAS-026  building  upon  the  initial  version 
of  the  COBP  produced  by  SAS-002.  This  new  COBP 
is  a  synthesis  of  decades  of  expertise  from  various 
countries  and  hundreds  of  analyses.  The  COBP  was 
developed  using  a  set  of  case  studies  to  test  out  the 
varied  advice  and  guidance  received,  and  incorporates 
feedback  from  users  of  the  initial  version.  Lastly,  SAS- 
039  provided  a  peer  review  of  the  final  draft  product. 

The  earlier  version  focused  on  the  analysis  of  ground 
forces  at  a  tactical  echelon  in  mid  to  high  intensity 
conflicts.  Consequently,  the  initial  version  of  the  COBP 
did  not  completely  address  the  full  range  of  important 
issues  related  to  C2.  In  developing  this  new  version  of 
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the  COBP,  SAS-026  explicitly  focused  upon  OOTW, 
the  impact  of  significantly  improved  information  related 
capabilities,  and  their  implications  for  military 
organisations  and  operations.  In  addition,  SAS-026  was 
cognisant  of  the  fact  that  NATO  operations  are  likely  to 
include  coalitions  of  the  willing,  which  might  involve 
Partnership  for  Peace  (PfP)  nations,  others  partners 
outside  of  NATO,  international  organisations,  and 
NGOs.  NATO  operations  may  also  be  “out  of  area.” 

Feedback  from  users  of  the  original  COBP  also 
identified  a  number  of  ways  in  which  the  original  COBP 
could  be  improved.  These  areas  were  addressed 
during  the  development  of  this  version  of  the  COBP. 

Cost  analyses  continue  to  be  explicitly  excluded  for 
two  reasons.  First,  cost  analysis  is  a  mature  discipline 
that  experienced  operational  analysts  already  practice. 
Flence,  C2  issues  are  not  unique  in  the  arena.  Second, 
most  nations  have  already  developed  approaches  to 
cost  analysis  and  cost  effectiveness  that  are  consistent 
with  their  national  approaches  to  accounting.  Because 
these  national  practices  differ  among  NATO  members, 
no  single  approach  would  be  appropriate. 

As  this  COBP  is  being  drafted,  novel  experiments  with 
new  information-related  capabilities,  particularly 
networking  and  ways  to  accomplish  their  assigned  tasks 
abound.  Advances  in  technology  are  expected  to 
continue  at  an  increasing  rate  and  spur  both  sustaining 
and  disruptive  innovation  in  military  organisations.  It  is 
to  be  expected  that  this  COBP  will  need  to  be 
periodically  revisited  in  light  of  these  developments. 
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Overview  of  COBP  Assessment 
Philosophy 

The  COBP  assumes  that  the  objective  of  a  C2  system 
is  to  exercise  control  over  its  environment,  through 
either  adaptive  or  reactive  control  mechanisms,  or 
some  combination  of  those  two  approaches.  This 
provides  the  context  and  point  of  departure  for  the 
assessment  of  C2. 

Analysis  of  C2  should  consider  all  the  relevant  actors, 
military  command  levels,  and  functions  involved  and 
should  investigate  issues  of  integration  across 
disparate  organisations,  military  command  levels,  and 
functional  domains  over  time.  Consideration  should 
also  be  given  to  the  robustness  and  security  of 
information  systems  and  to  human  computer  interface 
issues.  Human  behavioural,  physiological,  and 
cognitive  factors,  along  with  organisational  and 
doctrinal  issues,  must  be  considered  in  C2  analyses. 

C2  assessments  must  also  consider  a  range  of 
missions,  adversary  capabilities,  and  adversary 
behaviours.  Moreover,  it  must  be  understood  that 
adversaries  will  use  asymmetric  tactics  and  techniques 
to  deny  or  exploit  differences  in  technology,  force  size, 
information  systems,  or  cultural  factors.  Hence, 
scenarios  and  analyses  that  deal  with  an  appropriate 
set  of  all  these  dimensions  should  be  considered  in 
either  the  main  research  design  or  in  the  excursions 
to  assess  risks  and  uncertainty. 

The  evaluation  of  C2  issues  depends  in  important  ways 
on  both  distinguishing  and  linking  dimensional 
parameters,  measures  of  performance,  measures  of 
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C2  effectiveness,  and  measures  of  force  and  policy 
effectiveness.  Modelling  and  other  tools  must  be 
designed  to  support  this  requirement. 

Tools  and  data  used  in  C2  analysis  should  conform  to 
good  OA  processes  and  practices  and,  to  the  extent 
feasible,  should  be  subject  to  Model  Verification, 
Validation,  and  Accreditation  (VV&A)  and  to  Data 
Verification,  Validation,  and  Certification  (VV&C). 

Interoperable  analytical  infrastructures  (e.g.  data 
dictionaries,  glossaries,  models,  tools,  data  sets)  are 
necessary  to  facilitate  the  efficient  proliferation  and 
reuse  of  study  results  and  data  within  the  broader 
interdisciplinary  research  community. 

Because  the  complexity  of  C2  and  the  requirements 
for  its  analysis  are  often  underestimated  by 
decisionmakers,  a  continuing  dialogue  between 
analysts  and  those  decisionmakers  is  necessary  both 
to  scope  the  problem  properly  and  to  ensure  that  the 
analytic  results  are  properly  understood.  Part  of  this 
process  includes  performing  sensitivity  analyses  and 
other  common  practices  designed  to  ensure  the  validity 
and  reliability  of  the  results. 

Changes  to  C2  systems  will  often  lead  to  changes  in 
military  concepts,  command  approaches,  doctrine, 
Tactics,  Techniques,  and  Procedures  (TTP),  and  related 
factors,  which  must  also  be  considered  in  the  analysis. 

Current  State  of  Practice  in  C2  Analysis 

Assessment  of  C2  issues  typically  employs  classic 
tools  of  OA.  Relatively  few  specialised  tools  and 
methods  have  been  developed  for  C2.  Moreover,  those 
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specialised  tools  generated  to  deal  with  the  unique 
aspects  of  C2-focused  research  are  generally  not  as 
well  understood  as  those  used  in  more  traditional 
warfare  modelling  domains.  C2  analysts  will  often  find 
themselves  having  to  develop  tools  and  approaches 
appropriate  for  their  research  agendas.  However,  a 
general  analytic  process  can  be  identified  that  will 
enhance  the  likelihood  that  an  OA  analyst  can  conduct 
successful  analyses. 

Organisation  of  the  COBP 

This  COBP  is  organised  into  four  themes.  The  first 
theme  deals  with  study  dynamics,  problem  formulation, 
and  the  development  of  a  solution  strategy.  The 
second  theme  identifies  and  discusses  in  depth  the 
essential  elements  of  assessment:  measures  of  merit, 
scenarios,  human  and  organisational  issues,  data,  and 
tools.  The  third  theme  addresses  issues  related  to  risk 
and  uncertainty  while  the  final  theme  describes  the 
range  of  assessment  products. 

This  represents  a  significant  enhancement  of  the  initial 
COBP.  In  particular  the  first,  third  and  fourth  themes 
were  not  treated  in  detail  in  the  initial  version  of  the 
COPB.  In  addition,  material  has  been  added  to  the 
second  theme  to  address  the  unique  assessment 
challenges  associated  with  OOTW. 

Brief  History  of  SAS-026 

SAS-026  builds  upon  almost  a  decade  of  work  that 
began  with  the  formation  of  the  Ad  Hoc  Working  Group 
on  the  Impact  of  C3I  on  the  Battlefield  by  Panel  7  of  the 
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NATO  Defence  Research  Group  in  1 991  to  assess  the 
state  of  the  art  in  C2  analysis.  Based  on  the 
recommendations  of  the  Ad  Hoc  Working  Group,  Panel 
7  constituted  Research  Study  Group-19  (RSG-19)  to 
address  issues  of  methodology,  measures  of  merit,  and 
tools  and  analysis.  The  panel  also  addressed  issues  of 
improving  a  nation’s  capability  to  examine  C2  acquisition 
and  decisionmaking.  At  the  October  1995  RSG-19 
planning  meeting,  the  group  determined  that  the  primary 
product  of  RSG-19  was  to  be  a  Code  of  Best  Practice 
for  assessing  C2.  As  part  of  selected  RSG-1 9  meetings, 
workshops  would  be  conducted  to  support  the 
development  of  the  major  sections  of  the  COBP. 
Workshops  were  conducted  on  Measures  of  Merit 
(Canada),  Scenario  Development  (Netherlands),  C3I 
Systems,  Structures,  Organisations,  and  Staff 
Performance  Evaluations  (Norway),  and  Models  Used 
for  C3  Systems  and  Analysis  (US/UK).  Representatives 
from  the  nations  in  parentheses  took  the  lead  in 
organising  the  workshops  and  summarising  the  results. 
The  minutes  of  the  workshops  provide  further 
illustrations  of  the  techniques  presented  in  the  COBP. 

At  the  October  1996  meeting,  the  group  took  up  a 
request  by  Panel  7  to  conduct  a  symposium  on 
modelling  and  analysis  of  C3I,  which  was  scheduled 
at  the  July  1997  meeting  for  January  1999.  This 
symposium  was  a  forum  for  presentation  and 
discussion  of  the  COBP  and  related  topics. 

At  the  July  1997  meeting,  in  response  to  a  query  by 
Panel  7,  the  group  discussed,  acknowledged,  and 
agreed  on  the  need  for  a  follow-on  group  to  SAS-002. 
An  exploratory  group  on  organisational  change  (SAS- 
E05)  was  formed  to  recommend  a  way  ahead. 
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SAS-E05  recommended  the  formation  for  a  follow-on 
activity  to  SAS-002  to  accomplish  four  objectives: 

•  Demonstrate  and  assess  the  initial  version  of 
the  COBP; 

•  Revise  and  extend  the  COBP; 

•  Identify  research  areas;  and 

•  Facilitate  the  adoption  of  the  COBP. 

The  SAS  panel  concurred  in  May  1 999  and  approved 
the  formation  of  SAS-026,  which  began  its  2  1 /2-year 
plan  of  work  in  a  symposium  in  January  2000. 

Chapter  1  Acronyms 

C2  -  Command  and  Control 

C3(l)  -  Command,  Control,  Communications  (and 
Intelligence) 

C4(l)  -  Command,  Control,  Communications,  and 
Computers  (and  Intelligence) 

C4ISR- Command,  Control,  Communications,  Computers, 
Intelligence,  Surveillance  and  Reconnaissance 

CIS  -  Command  Information  Systems 

COBP  -  Code  of  Best  Practice 

DP  -  Dimensional  Parameters 

MoCE  -  Measures  of  C2  Effectiveness 

MoFE  -  Measures  of  Force  Effectiveness 

MoM  -  Measures  of  Merit 
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MoP  -  Measures  of  Performance 

MoPE  -  Measures  of  Policy  Effectiveness 

NGO-  Non  Governmental  Organisations 

OA  -  Operational  Analysis 

OOTW  -  Operations  Other  Than  War 

PfP  -  Partnership  for  Peace 

PVO  -  Private  Volunteer  Organisations 

RSG-19  -  Research  Study  Group-19 

TTP- Tactics,  Techniques,  and  Procedures 

W&A-  Verification,  Validation,  and  Accreditation 

W&C- Verification,  Validation,  and  Certification 
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'As  an  illustration,  General  Colin  Powell,  then  Chairman,  Joint 
Chiefs  of  Staff,  summarised  the  mission  in  Desert  Storm  by 
stating  that  “First,  we  will  cut  off  the  enemy  and  then  we  will  kill 
it.”  (Pentagon  Briefing,  Wednesday  January  23,  1 991 .) 

2As  an  example,  the  US  Congress  has  continually  sought  to 
impose  an  arbitrary  deadline  for  US  forces  to  withdraw  from 
Bosnia. 

3Mission  capability  packages  include  all  of  the  elements 
necessary  for  an  operation.  (Alberts,  1996). 
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Preparing  For 
Success: 
Assessment 
Participants, 
Relationships, 
and  Dynamics 


For  hypotheses  ought... to  explain  the 
properties  of  things  and  not  attempt  to 
predetermine  them  except  in  so  far  as 
they  can  be  an  aid  to  experiments. 

— Isaac  Newton  (1687) 

We  have  run  out  of  Money — now  we 
have  to  think. 

— Ernest  Rutherford  (1 871-1 937) 


Overview 


This  chapter  is  organised  into  three  parts.  The  first 
discusses  the  roles  played  by  the  significant  players 
associated  with  a  Command  and  Control  (C2) 
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assessment  and  how  these  roles  affect  the  design  and 
conduct  of  the  assessment.  The  second  part  identifies 
the  major  phases  of  a  C2  assessment  and  their  iterative 
nature.  The  concluding  section  addresses  the  subject 
of  professional  ethics  and  standards  of  conduct. 

Assessment  Participants 

Like  their  subject,  the  organisation  of  C2  studies 
involves  complex  interrelationships.  It  is  crucial  for  the 
analytical  team  to  establish  which  individuals  and 
organisations  are  involved  at  an  early  stage  of  the 
study.  It  is  prudent  for  the  analytical  team  to  map  the 
roles  described  below  onto  the  individuals  and 
organisations  involved  and  to  understand  their 
interrelationships.  An  example  of  such  a  mapping  is 
at  Figure  2.1.  Appendix  1  to  this  chapter  provides  a 
brief  explanation  of  the  organisations  involved. 

Due  to  the  dynamic  nature  of  such  projects,  those 
involved  should  not  be  surprised  if  the  nature  of  the  teams 
involved  might  have  to  expand  or  change  with  time. 

Assessment  Team 

The  assessment  team  is  working  for  the  sponsor  or 
client  (sponsor).  The  team  consists  of  a  senior  team 
leader  (who  may  also  be  referred  to  as  the  project 
manager),  a  core  set  of  analysts,  subject  matter 
experts  including  military  officers,  and  supporting  staff 
who  are  working  on  the  study  on  a  day  to  day  basis. 
The  team  provide  the  legitimacy  and  authority  for  the 
study.  The  sponsor  will  provide  the  terms  of  reference, 
access  to  needed  information,  and  identify  the  desired 
products.  It  is  important  for  the  analytical  team  to 
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understand  exactly  why  the  sponsor  wants  the  study 
and  what  the  sponsor  wants  to  do  with  the  results. 

Decisionmakers  or  Problem  Owners 

The  decisionmakers  are  the  individuals  or 
organisations  that  are  expected  to  make  decisions 
wholly  or  partially  based  on  the  output  or  findings  of 
the  study.  If  there  is  no  decision  to  be  made  (i.e.  this 
is  an  exploratory  study)  then  the  decisionmakers  could 
be  referred  to  as  problem-owners.  It  is  important  for 
the  assessment  team  to  understand  exactly  what  type 
of  assessment  the  decisionmakers  want  the  study  to 
support.  The  decisionmakers  may  or  may  not  be  in 
command  of  or  part  of  the  sponsor’s  organisation. 
Complex  problems  may  arise  when  the  decisionmaker 
is  several  steps  laterally  away  in  the  organisation  from 
the  sponsor  and  study  team. 

Stakeholders 

The  stakeholders  are  the  persons  or  organisations  that 
are  directly  or  indirectly  affected  by  the  study  outcome. 
Stakeholders  may  also  play  other  roles.  The 
assessment  team  must  be  aware  of  the  potential  for 
conflict  when  the  stakeholders  do  not  include  the 
sponsor  or  decisionmaker.  Complex  problems  may  arise 
in  the  provision  of  data  for  the  study,  as  it  is  the 
stakeholders  who  may  have  to  provide  the  data,  set 
the  security  or  releasibility  of  that  data  (and  hence  the 
study),  and/or  agree  that  the  data  are  representative. 
For  these  reasons  it  is  essential  that  the  analysts 
establish  a  working  relationship  with  the  stakeholders 
early  in  the  process. 
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Bill  Payer 

The  bill  payer  is  the  organisation  or  individual  official 
paying  for  the  study.  It  is  important  for  the  assessment 
team  to  know  the  level  of  resources  available.  Bill 
payers  will  normally  have  a  direct  interest  in  the 
outcome  of  the  study  and  may  be  one  or  more  of  the 
other  players.  Contractual  authorities  have  the  legal 
authority  to  let  contracts  on  behalf  of  the  bill  payer. 

Existing  Study  Teams 

The  assessment  team  must  be  aware  of  and  sensitive 
to  the  existence  of  teams  in  other  related  study  areas. 
Should  such  teams  exist,  the  assessment  team  should 
endeavour  to  exploit  the  work  done  and  available  skills 
and  techniques.  Such  external  teams  may  also  be 
appropriate  for  membership  in  peer  reviews. 

Future  Study  Teams 

The  assessment  team  must  be  aware  of  and  sensitive 
to  the  needs  of  future  analyses  and  assessments.  Data 
collection,  method  documentation,  and  the  archiving 
of  data,  methods,  models  and  results  are  fundamental 
responsibilities  of  all  professional  analysts.  Method  and 
data  should  be  (as  far  as  is  practicable)  disseminated 
and  published. 

Peer  Reviewers 

Outside  experts  brought  in  to  look  at  the  work  and 
provide  constructive  criticisms  are  called  peer 
reviewers.  Peer  review  teams  could  be  composed  of 
specialists  and  other  study  teams  in  related  subject 
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areas  and  should  include  representatives  from  all  key 
disciplines  in  the  assessment. 

Data  Providers 

Data  providers  are  the  individuals  and  organisations 
that  possess  data  and  information  useful  to  the 
assessment  team.  Many  of  these  will  be  stakeholders. 
The  motivation  to  provide  data  to  the  study  must  be 
developed  by  the  analytical  team  and  the  sponsor. 

Assumption  Providers 

Assumption  providers  are  the  individuals  or 
organisations  that  can  provide  “givens”  such  as  future 
doctrine,  performance  data,  force  mixes, 
organisational  structures,  and  scenarios.  Creation  of 
a  positive  relationship  with  these  organisations  is 
important  to  the  study. 

Data  Collectors 

In  some  C2  analyses,  where  data  must  be  extracted  from 
real  world  experiences,  exercises,  experiments,  and 
wargames,  teams  of  data  collectors  and  subject  matter 
experts  will  be  required.  The  identification  of  people  with 
the  appropriate  background  and  training  as  data 
collectors  is  an  important  element  of  such  studies. 


26 


NATO  Code  of  Best  Practice  for  C2  Assessment 


Relationships  Among  Participants 
and  the  Conduct  of  the  Assessment 

Relationships 

Figure  2.1  illustrates  how  complicated  the  participant 
roles  and  relationships  can  be  in  a  real  C2  assessment. 
This  particular  figure  represents  the  organisations 
involved  in  the  recent  Immediate  Reaction  Task  Force 
(Land)  (IRTF(L))  C2  Concept  evaluation  that 
completed  at  the  end  of  2001  (Candan  &  Lambert, 
2002).  Appendix  1  to  this  chapter  provides  a  brief 
explanation  of  the  organisations  involved.  Although  not 
all  projects  will  be  this  complex,  many  important  C2 
assessments  will. 

Through  the  prudent  act  of  mapping  the  roles  of  the 
participants  of  the  study,  the  potential  conflicts  of 
interest  and  complex  interactions  can  be  identified. 
One  method  to  mitigate  these  is  to  present  or  conduct 
this  activity  openly  and  discuss  with  all  involved  so 
that  all  potentially  affected  participants  are  aware  of 
the  possibility  of  future  conflict  and  the  fact  that  all 
participants  fall  into  one  or  more  roles  within  a  project. 

In  the  event  of  conflict  with  other  participants  in  the 
project  the  assessment  team  should  address  the 
issues  in  a  neutral  and  independent  manner. 
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Figure  2.1 .  Example  of  the  Mapping  of  Roles  onto 
Players — for  a  Complex  C2  Project  (the  Evaluation  of 
Immediate  Reaction  Task  Force  (Land)  C2  Concept) 


Understanding  the  Context  of  the 
Assessment 

The  relationship  among  the  assessment  team,  the  key 
sponsor,  and  the  stakeholders  is  of  paramount 
importance  and  perhaps,  more  than  any  other  single 
factor,  will  influence  the  course  and  success  of  the 
effort.  Accordingly,  adequate  attention  needs  to  be  paid 
to  understanding  the  situation  facing  the  key  sponsor 
and  stakeholders  as  much  as  the  subject  under  study. 

The  assessment  team  should  be  aware  that  the 
different  participants  will  have  divergent  perspectives 
and  may  have  divergent  agendas. 

Therefore  the  assessment  team  should  understand  the 
background  of  the  individuals  involved,  their 
organisational  settings,  roles  and  responsibilities,  their 
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history,  and  their  current  situation.  Contact  with  analysts 
who  have  worked  with  this  sponsor  and  review  of  prior 
analyses  for  this  sponsor  facilitate  this  objective. 

It  is  good  practice  to  build  and  maintain  long-term 
relationships  with  the  sponsor  and  stakeholder 
organisations.  This  will  yield  substantial  dividends  in 
the  form  of  easier  communication,  greater  trust,  and 
stronger  support. 

A  Continuing  Dialogue 

It  is  important  that  a  dialogue  with  the  sponsor  and 
stakeholders  is  maintained  by  the  assessment  team 
throughout  the  study.  As  there  is  no  single  “language” 
that  will  describe  the  study  problem,  it  is  important  to 
spend  time  at  the  beginning  of  a  study  to  establish  a 
common  “language”  that  both  the  assessment  team 
and  the  sponsor  and  stakeholder  can  understand.  This 
point  may  seem  obvious  in  a  NATO  setting  in  which 
the  participants  speak  many  different  natural 
languages.  However,  it  is  equally  important  in  a  single 
language  setting  because  common  words  and  phrases 
have  different  meanings  for  different  organisations, 
services,  and  even  individuals  within  a  single 
organisation.  Regular  meetings  and  contact  will 
minimise  misunderstandings.  From  a  professional 
point  of  view,  Operations  Research  (OR)  and 
Operations  Analysis  (OA)  analysts  will  always  wish  to 
inform  the  sponsor  and  stakeholders  of  key 
developments  and/or  challenges  as  the  study  unfolds. 
Regular  and  routine  interactions  need  to  be  built  into 
the  project  plan.  If  there  are  multiple  sponsors  and 
stakeholders  and  other  key  actors,  the  assessment 
team  should  try  to  meet  them  jointly,  particularly  when 


Chapter  2  29 


decisions  need  to  be  made.  Separate  meetings  will 
often  lead  to  inconsistent  guidance  and  will  place  the 
assessment  team  in  a  position  of  trying  to 
accommodate  differing  interests. 

The  development  of  a  collectively  agreed  upon  Study 
Glossary  that  captures  the  definitions  of  words,  phrases 
and  acronyms  used  in  the  study  is  a  useful  tool. 

Terms  of  Reference 

A  good  “term  of  reference”  covers  goals,  scope, 
products,  schedule,  and  resources.  These  will 
determine  the  focus  of  the  assessment  and  establish 
limits  or  freedoms  granted  to  the  assessment  team 
within  the  sponsor’s  and  stakeholder’s  organisations. 
Letters  of  introduction  and  instructions  to  actors  within 
the  sponsor  and  stakeholder  organisations  may  also 
be  useful. 

Understanding  How  the  Output  of  the 
Study  Will  be  Used 

It  is  important  to  know  at  an  early  stage  in  the  project 
what  the  products  of  the  study  are  to  be  used  for  by 
the  sponsor  and  stakeholder  organisations.  The 
expected  end  product  will  set  the  tone  and  relative 
importance  of  the  project  in  the  eyes  of  the  sponsor, 
stakeholder,  and  other  actors.  The  assessment  team 
needs  to  establish  and  understand  the  products 
needed  or  desired  by  the  sponsor  and  stakeholder. 
For  example,  a  study  could  be  used  to  create  a 
significant  impact  on  the  stakeholder’s  domain,  gain  a 
greater  understanding  of  the  issues,  produce  an 
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improved  capability  to  perform  future  work,  and/or 
make  contributions  to  the  body  of  knowledge. 

Budget 

The  sponsor  will  have  limited  resources  with  a  study 
budget  in  mind.  When  the  sponsor’s  resources  are 
limited  to  a  level  below  what  is  required  to  support  a 
quality  study,  the  assessment  team  will  need  to 
suggest  strategies  to  address  the  shortfall.  Alternative 
approaches  include  decomposing  the  problem  and 
only  undertaking  the  core  part  of  the  study  that  is 
affordable,  linking  the  sponsor  to  other  actors  that  have 
an  interest  in  the  same  or  a  similar  problem  and  who 
could  contribute  resources,  and/or  stretching  the 
project  over  a  longer  time  so  that  resources  from  future 
budget  cycles  become  available.  In  developing 
strategies  that  involve  doing  only  a  part  of  the  study  to 
satisfy  budget  constraints,  care  must  be  taken  to 
ensure  that  the  product  that  will  be  produced  provides 
a  meaningful  answer  or  contribution  and  does  not 
depend  upon  a  follow-on  effort  that  may  or  may  not 
be  funded. 

It  may  take  a  complete  iteration  of  the  assessment 
phases  of  the  project  to  establish  the  complete  scope 
of  the  project  and  the  resources  required.  Therefore  it 
is  good  practice  in  large  C2  projects  to  allow  the 
assessment  team  to  perform  a  rapid  first  pass  of  all 
the  phases  of  the  project  to  help  establish  the  budget 
required.  This  is  contrary  to  the  usual  practice  of  setting 
the  budget  in  stone  immediately  following  the  initial 
Problem  Formulation  or  Solution  Strategy  phase  (see 
Section  2-E  below). 
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Relationships 

Figure  2.1  illustrates  how  complicated  the  participant 
roles  and  relationships  can  be  in  a  practical  C2 
assessment.  This  particular  figure  represents  the 
organisations  involved  in  the  recent  Immediate 
Reaction  Task  Force  (Land)  (IRTF(L))  C2  Concept 
evaluation  that  completed  at  the  end  of  2001  (Candan 
&  Lambert,  2002).  Although  not  all  projects  will  be  this 
complex,  many  important  C2  assessments  will  be. 

Building  an  Assessment  Team 

Skills  Available  to  the  Assessment  Team 

Following  initial  problem  formulation  (Chapter  3),  the 
precise  skills  and  experience  required  by  the 
assessment  team  will  need  to  be  established.  Typically, 
the  assessment  team  will  need  to  be  interdisciplinary. 
The  wide  range  of  skills  and  experience  required  can 
be  allocated  between  a  full-time  core  team  and  a 
collection  of  consultants  or  part-time  team  members. 
As  an  example,  an  ideal  breakdown  of  the  skills  available 
to  the  assessment  team  involved  in  the  evaluation  of 
the  Immediate  Reaction  Task  Force  (Land)  C2  concept 
study1  is  given  below: 

Skills:  Core  Team 

•  Project  management; 

•  OR/OA  skills:  simulation,  wargaming, 
mathematical  programming,  database  creation 
and  management,  brainstorming  and  problem 


32 


NATO  Code  of  Best  Practice  for  C2  Assessment 


structuring,  scientific/military  report  writing/ 
editing; 

•  Cross  military  experience — i.e.,  OR/OA 
personnel  with  military  experience  or  military 
personnel  with  OR/OA  knowledge; 

•  Organisational  theory;  and 

•  Data  collection  (e.g.,  questionnaire  and  form 
design). 

Skills  and  Experience:  Consultants  and  Part-time 
Team  Members 

•  Military  (or  access  to  practical  experience  of 
problem  under  study); 

•  Training  and  exercise  planning  (if  an  exercise  is 
to  be  used  as  a  vehicle  for  the  study); 

•  Communications  and  information  systems 
specialists  for  the  systems  of  the  organisation 
under  study; 

•  Human  computer  interface  expertise; 

•  Operations  Other  Than  War  (OOTW)  related 
issues  (e.g.,  C2/Headquarters  [HQ],  media,  civil- 
military  cooperation — theory  and  practice); 

•  Social  scientists  (e.g.,  political,  psychological, 
economic,  cultural,  legal); 

•  Military  history; 

•  Command  experience; 

•  Deployment  analysis; 

•  Intelligence/threat/area  of  operations  expertise;  and 
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•  Legal/contracts/administration  expertise. 

As  another  example  of  the  skills  required  for  C2 
Assessment  Studies,  the  skills  required  to  provide  OR 
support  to  C2  elements  (such  as  OR/OA  support  to 
an  operational  HQ)  is  also  analogous,  as  illustrated  in 
Table  2.1  (RTO,  1999). 


Key  Af*A* 

Knowledge  Capabilities .  and  Smlls  Required 

Analytical 

pTctiablt  t  theory  Management  taels 

Parametric  and  norvpararredtic  Organisation  theory 

statabc*  information  management 

Force  and  systems  modeling  Process  engineering 

T ra rational  OP  Icc hr* cues'  Systems  dynamics 

Ded&cn  toos 

MiUary 

Understand  ml  nary  staff  Ovii  mltary  a*avs 

Staff  organisation  Situabcn  appreciabor 

Inter  Porsonal 

Interpersonal  skils  Consultant  sKils 

Instructor  skits  Wrung  ard  presen  lab  or.  Bulb 

Software 

Competencies 

Word  processing  Project  nysnagcmenl 

Spreadsheets  Mail  systems 

Pfwentabon  Network  systems 

Databases  World  wide  wed 

Mopprvg  Programming  language 

Plus  other  specialty  systems 

Harctoare 

Expertise 

Computers  Peripheral  devices 

Pnnters  Networks 

Scanners 

Table  2.1.  Knowledge,  Capabilities  and  Skills  Needed 
by  OR/OA  Cell  Team  Members 


Background  of  the  Assessment  Team 

Building  a  C2  assessment  team  with  this  full  breadth 
of  knowledge,  capabilities,  and  skill  requires  a  long¬ 
term  commitment  by  the  mother  OR/OA  organisation 
to  prepare  a  corpus  of  potential  team  members  through 
recruitment,  education,  training,  and  opportunities  for 
appropriate  field  experience. 

Following  the  identification  of  the  skills  required  for 
the  team,  those  analysts  made  available  for  the  team 
should  ensure  that  they  leave  a  basic  understanding 
of  the  military  fields  under  consideration.  Gaps  in 
experience  should  be  rapidly  filled  through  background 
reading,  short  courses,  field  experience,  or  additional/ 
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alternative  analysts  with  the  appropriate  experience 
and  skills. 

Forming  the  Assessment  Team 

In  a  study  that  involves  dispersed  and  disparate 
organisations  and  teams,  the  effort  to  command  and 
control  the  study  group  must  be  recognised  and  effort 
and  time  built  into  the  study  plan.  This  can  be,  for 
example,  through  maintenance  of  distributed  working 
environments  such  as  web  portals,  information 
campaign  material  and  travel  time  to  meetings.  In  these 
cases  the  senior  team  leader  will  revert  to  a  role  more 
akin  to  a  project  manager. 

It  is  one  thing  to  assemble  a  group  of  people,  quite 
another  to  forge  them  into  a  coherent  effective  team. 
Sufficient  time  and  a  facilitating  process  should  be  built 
in  to  the  project  plan  for  the  group  of  individuals  to 
coalesce  into  a  team. 

Interdisciplinary  Assessment 
Team  and  Outside  Relations 

It  follows  that  C2  analysis,  particularly  for  OOTW  issues, 
should  be  done  by  an  interdisciplinary  assessment 
team.  Experienced  analysts  know  that  their  work  owes 
success  in  no  small  measure  to  efficient  working 
relationships  within  the  assessment  team  and  with  the 
customer  of  analysis.  Building  good  working 
relationships  among  representatives  of  different 
scientific  cultures,  such  as  OR/OA  and  IT  analysts 
grounded  in  (hard)  physical  sciences  and  mathematics 
on  one  hand  and  (soft)  social  scientists  on  the  other, 
requires  sufficient  mutual  understanding  of 
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methodologies  and  tools.  In  fact,  differences  in  scientific 
cultures  can  outweigh  differences  in  natural  cultures 
provided  that  all  members  of  the  assessment  team  have 
sufficient  command  of  a  common  language.  Therefore, 
in  addition  to  leadership  and  project  management  skills, 
the  head  of  the  assessment  team  must  have  a  good 
general  idea  of  the  current  state  of  all  disciplines  involved 
in  order  to  compose  an  efficient  team  and  facilitate 
interdisciplinary  cooperation  throughout  the  analysis. 

Good  personal  and  working  relationships  with  the 
customer  of  the  analysis  are  essential  for 
understanding  every  aspect  of  the  problem  and  being 
able  to  arrive  at  a  problem  structure  and  solution 
strategy  that  meets  the  customer’s  immediate  needs 
in  the  light  of  the  strategic  objectives  of  the  respective 
OOTW.  Knowing  the  customer’s  position  in  the 
command  hierarchy  and  the  degree  of  influence  he/ 
she  wields  through  informal  relationships  over 
stakeholders  and  actors,  the  co-operation  of  which 
might  be  essential  for  an  implementation  of  analysis 
results,  and  understanding  the  respective 
consequences  associated  with  alternative  solutions  is 
important  for  assessing  their  acceptability  and 
organisational  risk. 

It  is  equally  important  for  the  assessment  team  to 
establish  working  relationships  with  the  potential 
subjects  of  study  in  the  early  stages.  This  is  essential 
for  capturing  the  nature  and  problem  relevance  of  formal 
and  informal  relations  between  all  organisations,  groups, 
and  individuals  that  are  subjects  of  the  study,  finding 
out  about  their  motivations  and  agendas,  and  eliciting 
firsthand  information  that  is  critical  for  solving  the 
problem  such  as  their  capabilities  and  the  conditions 
attached  for  their  employment.  However,  the  analyst 
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should  be  careful  not  to  allow  this  effort  to  gain  greater 
understanding  of  the  problem  to  introduce  bias. 

Assessment  Phases,  Process, 
and  Dynamics 

It  is  important  to  realise  that  all  of  the  elements  of  the 
C2  assessment  are  interrelated.  Hence  Problem 
Formulation,  Solution  Strategy,  Measures  of  Merit, 
Scenarios,  Human/Organisational  Factors,  Models/ 
Tools/Data,  and  products  are  all  interdependent. 
Figure  2.2  illustrates  the  major  phases  and  iterative 
nature  required  for  C2  assessments.  The  Assessment 
Process  diagram  was  the  most  difficult  thing  for  the 
SAS026  team  to  agree  upon.  In  essence  this  diagram 
is  at  the  heart  of  the  COBP  (Starr,  2001).  The 
remainder  of  this  chapter  discusses  the  key  points  in 
this  diagram. 


Figure  2.2.  C2  Assessment  Process 
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Problem  Formulation 

The  output  of  Problem  Formulation  (Chapter  3)  specifies 
the  “what”  of  the  assessment.  C2  studies  tend  to  be 
complex  and  feature  multiple  attributes,  some  of  which 
may  not  be  apparent  at  the  start  of  the  study.  Neither 
the  assessment  team  nor  the  sponsor  should  be 
surprised  if  the  issues  initially  presented  for  study  are 
replaced  by  other  issues  that  are  closer  to  the  underlying 
causes  of  the  initial  problem  or,  in  some  cases, 
symptoms  presented.  A  consequence  of  the  dynamic 
nature  of  problem  formulation  is  that  the  solution  strategy 
and  any  of  the  other  elements  of  the  assessment  may 
change  as  the  study  progresses.  Problem  formulation 
should  therefore  be  consciously  and  routinely  iterated 
during  a  study — especially  when  new  attributes  start  to 
appear.  As  a  minimum  an  iteration  should  occur 
immediately  following  the  establishment  of  the  initial 
solution  strategy  and  the  assessment  of  study  risk. 
Additionally,  the  sponsor  should  be  aware  that  the 
nature  of  the  assessment  team,  sponsor,  or  assumption 
provider  teams  might  also  have  to  expand  or  change 
with  time.  This  has  implications  for  planning,  budgeting, 
and  tasking. 

In  nearly  ail  C2  studies  the  assessment  team  will  study 
only  a  subset  of  the  whole  problem  space  due  to  the 
sponsor’s  sphere  of  interest.  This  fact  must  be 
recognised  by  the  assessment  team.  An  initial  study 
of  the  complete  problem  space  is  essential  to  establish 
this  realisation.  This  will  help  the  assessment  team  to 
understand  the  context  of  the  study  and  provide  advice 
to  the  sponsor  on  the  actual  underlying  causes  to  his 
problem  and  consequently  the  requirement  to  involve 
other  participants. 
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Solution  Strategy 

The  next  step  is  to  develop  a  Solution  Strategy 
(Chapter  4)  that  specifies  the  “how”  of  the  assessment. 
Arising  from  the  Solution  Strategy  agreed  upon  and 
adopted  by  the  sponsor  are  a  set  of  terms  of  reference 
(e.g.,  Statements  of  Work  [SOW]  for  contracts)  that 
will  determine  what  work  is  to  be  conducted,  the 
contractual  obligations,  deadlines,  and  resources. 
Although  these  must  be  established  as  an 
experimentation  campaign  plan3  and  study 
management  plan  (project  plan)  before  work  on  the 
project  begins  in  earnest,  flexibility  must  be  built-in  due 
to  the  iterative  nature  of  C2  assessment.  The 
assessment  team  must  be  aware  of  any  preconceived 
“solutions”  that  have  been  proposed  by  the  sponsor, 
stakeholders,  and/or  decisionmakers  and  explicitly 
deal  with  these  as  appropriate,  avoiding  another 
pressure  to  be  steered  in  a  particular  direction.  The 
assessment  team  must  note  if  its  results  are  being 
steered  in  a  particular  direction  and  follow  ethical 
behaviour  in  performing  the  study  (see  the  end  of  this 
chapter).  In  many  cases  a  risk-based  approach  to  C2 
assessments  can  usefully  complement  the  more 
traditional  cost-effectiveness  approach.  In  particular, 
this  helps  decisionmakers  to  deal  with  the  uncertainties 
of  the  real  problem. 

From  a  professional  point  of  view  analysts  should  always 
defer  the  selection  of  a  particular  method  until  the  problem 
has  been  formulated  and  a  solution  strategy  has  been 
defined.  Recognise  and  beware  of  “preconceived” 
solutions  that  could  influence  the  assessment. 
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Review 

Once  there  has  been  a  preliminary  formulation  of  the 
problem  and  development  of  a  solution  strategy,  it  is 
imperative  that  an  initial  review  be  conducted.  This 
review  should  be  conducted  from  multiple  perspectives 
(e.g.  with  respect  to  the  sponsor’s  initial  problem,  the 
feasibility  with  respect  to  resources  including  team 
skills  and  schedule,  soundness  of  the  proposed 
analytic  approach).  As  a  result  of  this  review,  changes 
will  usually  be  made  in  both  the  problem  formulation 
and  the  solution  strategy. 

Measures  of  Merit,  Scenarios,  and 
Human/Organisational  Factors 

At  this  stage  the  assessment  team  must  specify  the 
hierarchy  of  Measures  of  Merit  (MoM)  (Chapter  5), 
incorporate  and  identify  relevant  human  and 
organisational  factors  (Chapter  6),  and  specify  the 
appropriate  scenarios  (Chapter  7).  As  suggested  by 
the  diagram,  there  is  no  unique  sequence  for  doing 
these  tasks.  Iteration  is  required  to  ensure  that  these 
tasks  are  done  in  a  coherent,  consistent  fashion.  When 
ail  of  these  tasks  have  been  completed,  the  team  has 
specified  the  key  variables  to  the  necessary  level  of 
detail  with  adequate  considerations  for  assessment 
validity  and  reliability. 

When  developing  the  MoM  it  is  very  valuable  to  involve 
the  sponsor  in  establishing  the  linkages  between  the 
MoMs  and  the  hierarchy  of  MoM.  This  is  because  then 
the  sponsor  will  then  appreciate  the  dynamics  of  the 
problem  and  the  requirement  for  breadth  in  the  study. 
Although  a  full  set  of  MoM  must  be  derived  in 
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accordance  with  the  best  practice  noted  in  Chapter  5, 
the  MoM  should  be  prioritised  to  focus  on  providing 
support  to  the  objectives  of  the  study  and  be  practical 
and  cost  effective. 

When  selecting  appropriate  scenarios  it  is  good 
practise  to  utilise  scenarios  (if  they  exist)  from  a 
standard  set  of  scenarios  approved  for  use  within  the 
assessment  and  sponsor  organisation.  The  sponsor 
must  always  be  approached  for  approval  of  the 
scenarios.  It  is  bad  practice  to  design  a  scenario  to 
prove  a  point. 

Models,  Tools,  and  Data  Requirements 

The  next  step  is  to  iteratively  identify  the  methods  and 
tools  (Chapter  8)  and  data  (Chapter  9)  required  to 
perform  the  assessment.  One  of  the  major  challenges 
of  the  assessment  is  to  identify  and  gain  access  to 
models,  tools,  and  data  that  are  appropriate  for 
exploring  the  issues  of  interest.  The  challenges  come 
in  several  dimensions: 

•  First,  there  is  a  limited  set  of  tools  that  deal 
effectively  with  the  C2  dimension  of  the  problem. 

•  Second,  for  even  this  limited  set,  it  is  often 
difficult  to  access  and  modify  the  tools  to  reflect 
the  variables  of  interest. 

•  Third,  there  is  often  a  paucity  of  useful  data  and 
previously  validated  parameters. 

As  a  result  of  the  establishment  of  the  MoM  for  the 
study  and  the  data  that  underpins  those  MoM  and 
models,  a  data  collection  and  analysis  plan  should  be 
formulated.  The  sponsor  should  also  be  made  aware 
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of  the  difficulties  associated  with  getting  appropriate 
data,  cost  of  the  data  collection  and  analysis  plan,  and 
the  implications  to  the  study  if  the  required  resources 
are  not  set  aside  and  budgeted  to  collect,  collate, 
process,  and  analyse  the  data. 

Assess  Study  Risk 

At  this  point  in  the  process  the  assessment  team 
should  take  a  look  at  the  risks  and  uncertainties 
(Chapter  1 0)  associated  with  the  decisions  made  with 
respect  to  all  of  the  tasks  performed  to  date  (e.g., 
consistency  between  the  scenarios  and  the  data, 
models  and  availability  of  data,  tools  and  analysis). 
The  sponsor  must  be  made  aware  of  these  risks  and 
uncertainties  and  the  strategies  developed  by  the  team 
to  mitigate  them.  If  the  risks  associated  with  the 
successful  completion  of  the  study  are  perceived  as 
being  too  high,  the  solution  strategies  should  be 
revisited  and  adjusted  accordingly. 

Peer  Review 

When  the  risk  and  uncertainties  are  perceived  as 
manageable,  a  peer  review  should  be  conducted.  Peer 
reviews  are  not  used  enough  because  they  tend  to  be 
time-consuming,  seen  as  raising  costs,  or  perceived 
as  threatening.  In  addition,  research  teams  often  want 
to  perfect  their  results  and  methods  before  revealing 
them.  The  key  is  to  build  a  peer  review  into  the  study 
from  the  outset.  The  sponsor  should  be  informed  as 
to  the  importance  of  the  peer  review.  Peer  reviews 
should  be  built  into  the  budget  and  reviewers  invited 
to  look  at  the  terms  of  reference,  interim  products,  and 
draft  reports  so  that  they  are  knowledgeable  about 
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the  effort  and  motivated  to  support  the  project.  In  later 
stages  of  the  study,  the  peer  review  can  improve 
presentation  and  also  act  as  a  mechanism  to  make 
the  results  known  to  the  professional  communities. 
Over  time  the  assessment  team  should  develop  a 
relationship  with  high  quality  peers  and  use  them  as  a 
pool  of  reviewers. 

Peer  reviews  are  not  a  luxury  but  a  necessity. 

Conduct  of  the  Study 

At  this  point  we  are  in  a  position  to  execute  the 
assessment.  The  assessment  team  leader  should 
keep  a  study  notebook  or  journal  in  which  all 
assumptions  and  decisions  are  documented  so  that 
they  are  available  for  detailed  discussion.  Detailed 
administrative  records  need  to  be  kept  regarding  the 
data,  metadata,  models,  and  analytical  and 
documentation  tools.  This  will  enable  replication  of 
parts  of  the  C2  analysis  should  the  need  arise.  An 
effort  should  be  made  to  create  data  sets  (not  just  the 
project  results)  that  will  be  available  to  other 
researchers.  The  resources  required  to  make  such 
data  available  to  external  bodies  needs  to  be  made 
clear  to  the  sponsor.  The  conduct  of  the  study  will  not 
usually  be  linear.  It  should  be  anticipated  that  multiple 
iterations  will  be  conducted  and  that  lessons  learned 
from  initial  data  collection  and  analysis  efforts  will 
inform  subsequent  activities. 


Study  Products 

The  team  must  recognise  the  importance  of  presenting 
the  results  of  the  assessment  in  a  clear  and 
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comprehensive  manner,  taking  into  consideration  the 
style  of  the  decisionmaker  (Chapter  11).  It  is 
particularly  important  that  these  results  illuminate 
rather  than  obscure  the  uncertainties  associated  with 
the  assessment. 

Ethics 

Professional  operations  research  organisations,  such 
as  the  Military  Operations  Research  Society  (MORS), 
have  developed  professional  codes  of  ethics  (Annex 
C).  The  assessment  team  should  also  be  guided  by  a 
set  of  professional  ethics  and  standards  of  conduct  to 
ensure  the  integrity  and  quality  of  the  analysis.  This 
means  that  the  assessment  team  should,  inter  alia: 

•  Maintain  an  open  and  honest  dialogue  with  the 
sponsor  and  other  key  players  within  the  project 
in  order  to  minimise  misunderstandings; 

•  Ensure  that  C2  assessments  are  organised  and 
conducted  in  a  balanced  fashion  that  adequately 
identifies  and  represents  all  perspectives, 
options,  and  relevant  evidence; 

•  Inform  the  sponsor  and  other  key  players  of: 

•  any  constraints,  assumptions,  or 
circumstances  that  threaten  a  balanced 
assessment;  and 

•  the  risks  and  uncertainties  associated  with  the 
methods  and  data  used  in  the  project,  and 

•  strategies  to  minimise  the  risks. 


44 


NATO  Code  of  Best  Practice  for  C2  Assessment 


Chapter  2  Acronyms 

ACE  Resources- Allied  Command  Europe  Resources 
-  (Part  of  SHAPE) 

AF(N)  -  Regional  Command  (North) 

AMF(L)  -  ACE  Mobile  Force  (Land) 

ARRC  -  ACE  Rapid  Reaction  Corps. 

C2  -  Command  and  Control 

CDE  -  Concept  Development  and  Experimentation 

CPX-  Command  Post  Exercise 

FTX- Field  Training  Exercise 

HB(A)  -  UK  Historical  Branch  (Army) 

HQ  -  Headquarters 

IRTF(L)  -  Immediate  Reaction  Task  Force  (Land) 

JCSC  -  Joint  Sub-Regional  Command  South  Centre 

JCSE  -  Joint  Sub-Regional  Command  South  East 

JFCOM  -  US  Joint  Forces  Command 

KIBOWI  -  NL  Army  Exercise  Driver 

MND(C)- Multinational  Division  (Central) 

MoM  -  Measures  of  Merit 

MORS  -  Military  Operations  Research  Society 

NC3A  -  NATO  C3  (Consultation,  Command  & 
Control)  Agency 

NL  MOD  -  Netherlands  Ministry  of  Defence 
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OOTW  -  Operations  Other  Than  War 

OA- Operational  Analysis 

OR  -  Operations  Research 

PRL  -  SHAPE  Policy  Requirements  Land 

SACLANT  OA  -  Supreme  Allied  Command  Atlantic 
Operational  Analysis  Cell 

SFS  -  Strike  Force  South 

SHAPE  -  Supreme  HQ  Allied  Powers  Europe 

SOW  -  Statements  of  Work 

WPC  -  Warrior  Preparation  Center  (Ramstein  Germany) 

Chapter  2  References 

Research  and  Technology  Organization  (RTO)  SAS-003 
report  on  IFOR  and  SFOR  data  collection  &  analysis 
(6  AC/323(SAS)TP/1 1).  [Technical  Report],  (1999). 
Neuilly-Sur-Seine  Cedex,  France:  North  Atlantic 
Treaty  Organisation  (NATO)  RTO.  Available  from 
http://www.rta.nato.int/RDP.asp?RDP=RTO-TR- 
006 

Starr,  S.  H.  (n.d.).  (2001).  Lessons  recorded  from 
applying  the  North  Atlantic  Treaty  Organisation 
(NATO)  code  of  best  practice  (COBP)  for  C2 
assessment  to  operations  other  than  war 
(OOTW).  McLean,  VA:  MITRE  Corporation. 
Available  from  http://www.mitre.org/support/ 
papers/tech_papers_0 1/starr_nato/ 
starr_nato.pdf 


46 


NATO  Code  of  Best  Practice  for  C2  Assessment 


Candan,  U.  and  Lambert,  N.  J.  (2002).  Methods  used 
in  the  Analysis  and  Evaluation  of  the  Immediate 
Reaction  Task  Force  (Land)  Command  and 
Control  Concept.  TN  897.  NATO  C3  Agency, 
PO  Box  174,  2501  CD,  The  Hague,  The 
Netherlands. 

Appendix  1  to  Chapter  2:  Participant 
Mapping  of  the  Evaluation  of  the 
Immediate  Reaction  Task  Force 
(Land)  C2  Concept — An  Explanation 
of  Figure  2.1 

Background 

The  Immediate  Reaction  Task  Force  (Land)  (IRTF(L)) 
command  and  control  concept  was  proposed  in  1 998 
as  a  mechanism  to  modernise  the  ACE  Mobile  Force 
(Land)  (AMF(L)).  The  IRTF(L)  concept  is  predicated 
on  the  enlargement  of  AMF(L)  from  brigade  size  up  to 
division  size  with  a  single  streamlined  headquarters 
and  a  chain  of  command  using  embedded  mini-Task 
Group  HQ  cells.  This  was  evaluated  between  1999 
and  2001  as  a  test  case  for  the  NATO  Concept 
Evaluation  and  Experimentation  (CDE)  process  using 
a  series  of  FTX,  CPX,  wargames,  simulations  and 
historical  analyses. 

Assessment  Team 

In  the  case  of  the  IRTF(L)  study  the  assessment  team 
was  led  by  NC3A  OR  Division,  with  contracted  experts, 
analytical  and  military  support  from  KS  Consultants 
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and  UK  DERA.  Free  analytical  support  was  also  made 
available  at  peak  periods  from  US  JFCOM  and 
SACLANT  Operational  Analysis  (such  as  the  exercises 
of  the  experimentation  campaign). 

The  sponsor  was  SHAPE  Policy  and  Requirements  Land — 
who  were  tasked  with  the  evaluation  of  the  C2  Concept. 

It  was  clear  to  the  assessment  team  as  to  why  the 
sponsor  wanted  the  study — a  straightforward 
evaluation  of  the  military  utility  (to  NATO)  of  the  C2 
Concept.  However  at  the  end  of  the  study  the  results 
were  combined  with  other  issues,  and  decisions  were 
made  on  the  future  of  the  unit  under  study.  This  was 
something  that  was  not  foreseen  by  any  of  the 
participants  at  the  start  of  the  project. 

Decisionmakers  or  Problem  Owners 

The  sponsor’s  task  was  to  provide  advice  up  the  chain 
of  command  to  SACEUR  and  ultimately  the  Military 
Committee  on  the  efficacy  of  the  C2  Concept.  Although 
the  HQ  ACE  Mobile  Force  (Land)  was  the  subject  of 
the  study  it  was  also  party  to  any  decision  regarding 
its  own  future  modernisation.  It  is  commanded  directly 
by  SACEUR  via  SHAPE. 

SACLANT  CDE,  however,  was  not  in  the  command 
chain,  but  was  seen  as  a  decisionmaker  within  the 
context  of  the  study  as  it  was  interested  in  the 
experience  of  the  team  in  conducting  the  study  as  a 
test  case  to  illustrate  the  value  of  NATO  Centred  CDE 
to  the  Alliance. 
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Stakeholders 

The  ACE  Mobile  Force  (Land)  was  the  main 
stakeholder  as  it  was  the  subject  of  the  study.  As  a 
decisionmaker,  data  and  assumption  provider  and  also 
possible  member  of  a  future  study  team  it  was  in  a 
very  powerful  position,  and  was  approached  and 
treated  with  much  respect  by  the  assessment  team. 
After  a  shaky  start  (where  neither  side  was  sure  of  the 
other’s  intentions)  a  good  working  relationship  was 
established  over  the  period  of  the  project. 

The  Netherlands  MOD — in  the  form  of  the  Royal 
Netherlands  Army — was  the  provider  of  the  Command 
Information  System  (ISIS)  used  as  the  digitisation 
vehicle  for  the  evaluation  of  the  concept.  As  such  it 
was  directly  affected  by  the  exercise  program  used 
for  the  experimentation  and  also  the  future  of  the 
concept  and  AMF(L). 

The  Military  Committee  was  also  a  stakeholder, 
representing  the  Nations  of  NATO  that  contribute 
troops  and  staff  to  the  AMF(L),  and  these  nations  would 
be  directly  affected  by  any  decision  on  the  concept. 

ACE  Resources  at  SFIAPE  were  also  a  stakeholder  as 
they  were  required  to  sanction  and  organise  any  manning 
changes  proposed  for  the  HQ — including  the  temporary 
additional  manning  required  for  the  evaluation. 

Bill  Payer 

Monies  were  mostly  provided  from  the  slice  of  the 
NC3A  Scientific  Program  Of  Work  (SPOW)  controlled 
by  SHAPE  PRL.  In  the  initial  stages  of  the  project 
additional  monies  were  also  provided  by  SACLANT. 
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Monies  also  had  to  be  sought  from  the  SHAPE 
Exercise  budget  to  pay  for  movement  of  the  exercise 
observers  in  order  to  attend  the  exercises. 

Throughout  the  project  the  NC3A  was  the  contractual 
authority  to  let  contracts  on  behalf  of  the  bill  payer. 

Existing  Study  Teams 

An  extensive  literature  search  was  conducted  for  the 
study — with  the  majority  of  recent  references  occurring 
within  the  UK  and  US.  Exploratory  trips  (organised 
through  US  JFCOM)  to  US  Battle  labs  and  UK  facilities 
(through  UK  DERA)  revealed  the  current  state  of 
knowledge  with  respect  to  measuring  C2  performance 
in  exercises  and  evaluating  new  C2  concepts.  In 
response  to  this  the  data  collection  methodology  was 
based  initially  on  the  Fort  Leavenworth,  US  Army 
Research  Institute  ACCES  method. 

Future  Study  Teams 

It  was  identified  at  an  early  stage  that  there  could  be 
future  related  projects.  In  particular  those  relating  to 
expeditionary  and  initial  entry  forces.  The  probable 
NATO  organizations  that  could  be  involved  in  such 
studies  were  NC3A,  SHAPE  PRL,  AMF(L), 
Multinational  Division  (Central),  Strike  Force  South  and 
the  ACE  Rapid  Reaction  Corps.  Of  course  there  would 
probably  be  future  study  teams  within  the  nations — 
but  these  plans  are  not  visible  to  NAT O.  Consequently, 
as  the  assessment  team  was  very  likely  to  be  involved 
in  such  future  work;  all  data  was  archived  and  routinely 
written  up  and  published. 
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Peer  Reviewers 

The  assessment  team  were  not  able  to  arrange  a 
formal  Peer  Review  of  the  solution  strategy  adopted. 
This  mechanism  does  not  yet  exist  for  NATO  centred 
studies.  What  was  achieved  the  submission  of  the 
problem  to  the  SAS026  panel  as  an  example  for  testing 
the  coverage  of  the  revised  COBP.  This  yielded  some 
practical  advice  and  helped  the  assessment  team 
better  understand  the  dynamics  of  the  project. 

Data  Providers 

Most  of  the  data  for  the  evaluation  were  derived  from 
NATO  training  exercises  run  by  or  for  AMF(L), 
MND(C),  SFS  and  Joint  Sub  Regional  Commands 
South  East  and  South  Centre.  In  all  cases  relationships 
had  to  be  curried  by  the  assessment  team  and  sponsor 
to  allow  access  to  the  HQ  and  Exercise  Control  for 
the  exercise  observers,  and  for  background  materiel. 
In  two  cases  national  exercise  training  centres  and 
exercise  drivers  were  hired  by  the  assessment  team 
to  support  command  post  exercises  (Warrior 
Preparation  Center,  and  the  KIBOWI  exercise  driver). 
Historical  data  for  the  study  was  also  provided  from 
the  UK  Historical  Branch  (Army) — which  was 
approached  via  the  contracted  UK  members  of  the 
assessment  team. 

Assumption  Providers 

The  assessment  team  was  in  the  fortunate  position  to 
actually  be  one  of  the  assumption  providers — through 
NC3A’s  and  the  sponsor’s  involvement  in  the  NATO 
Defence  Requirements  Review.  The  owner  of  the  C2 
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Concept  however  remained  HQ  AMF(L)  itself,  and 
therefore  remained  the  authority  as  to  its  conceptual 
and  physical  implementation. 

Data  Collectors 

In  the  case  of  the  IRTF(L)  study,  data  was  largely 
extracted  through  observation  of  HQ  activities  during 
exercises  and  team-in-the-loop  wargames.  In  all  of 
these  exercises  Subject  Matter  Experts  (SME)  were 
used  to  observe  functional  and  cross  functional 
activities  in  the  HQ.  Most  of  the  military  SMEs  were 
provided  by  Regional  Command  AF  NORTH  and  its 
subordinate  commands  across  Allied  Command 
Europe  (ACE).  Additional  data  collectors  were  also 
provided  by  the  German  University  of  the  Federal 
Armed  Forces  and  US  JFCOM.  UK  DERA  provided 
military  analysts  to  lead  some  of  the  activities  involved 
in  capturing  the  HQ  processes. 


'As  developed  independently  by  the  SAS-026  panel  in  February 
2001  in  response  to  a  presentation  in  the  IRTF(L)  project.  In 
fact,  this  was  fairly  close  to  what  was  available  to  the  team. 
2Linear  programming,  dynamic  programming,  queuing  theory, 
inventory  control,  network  analysis  with  PERT,  game  theory  and 
simulation. 

3Required  if  the  C2  Assessment  makes  use  of  a  series  of  linked 
events  such  as  seminars,  wargames,  command  post  exercises 
(CPX),  field  training  exercises  (FTX),  etc. 


CHAPTER  3 


Problem 

Formulation 


First  find  out  what  the  question  is — 
then  find  out  what  the  real  question  is. 

— Vince  Roske 

Definition  of  Problem  Formulation 

Effective  problem  formulation  is  fundamental  to  the 
success  of  all  analysis,  but  particularly  in  Command 
and  Control  (C2)  assessment  because  the  problems 
are  often  ill-defined  and  complex,  involving  many 
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dimensions  and  a  rich  context.  Problem  formulation 
involves  decomposition  of  the  analytic  problem  into 
appropriate  dimensions  such  as  structures,  functions, 
mission  areas,  command  echelons,  and  C2  systems. 
Problem  formulation  is  an  iterative  process  that  evolves 
over  the  course  of  the  study.  It  is  essential  even  for 
small  studies  or  where  time  is  short — it  will  save  time 
later  and  help  ensure  quality. 

The  problem  formulation  phase  should  identify  the  context 
of  the  study  and  aspects  of  the  problem-related  issues. 

The  context  of  the  study  includes: 

•  Geopolitical  context  that  bounds  the  problem  space; 

•  Political,  social,  historical,  economic,  geographic, 
technological  environments; 

•  Actors; 

•  Threats; 

•  Aim  and  objectives  of  the  analysis,  including  the 
decisions  to  be  supported; 

•  Generic  C2  issues;1 

•  Relevant  previous  studies;  and 

•  Stakeholders  and  their  organizational  affiliation 
(including  both  stakeholders  of  the  problem  and 
stakeholders  of  the  study). 

The  aspects  of  the  problem  include: 

•  Issues  to  be  addressed; 

•  Assumptions; 
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•  High-level  Measures  of  Merit  (MoM); 

•  Independent  variables  (controllable  and 
uncontrollable); 

•  Constraints  on  the  values  of  the  variables 
(domain  and  range); 

•  Time  constraints  on  delivery  of  advice  to  the 
decisionmaker;  and 

•  Whether  this  is  a  single  decision  or  (possibly  one 
of)  a  chain  of  decisions  to  be  made  over  time. 

The  problem  is  not  formulated  until  the  assessment 
team  has  addressed  each  aspect  of  the  problem.  In 
simple  terms,  problem  formulation  can  be  seen  as  an 
iterative  process.  First,  the  team  must  identify  the 
variables  that  bound  the  problem  space.  Then  they 
must  determine  which  of  these  are  outputs  (dependent 
variables)  and  which  of  these  are  inputs  (independent 
variables).  The  team  proceeds  by  iterating  to  build  an 
understanding  of  how  these  relate  to  each  other.  It 
should  be  viewed  as  a  voyage  of  discovery.  In  most,  if 
not  all,  cases  of  C2  assessment,  the  knowledge 
domain  under  study  is  in  fact  a  system  characterised 
by  rich  interaction  and  feedback  among  all  the  factors 
or  variables  of  interest.  The  choice  of  dependent 
variables  results  from  a  clear  specification  of  the  issues 
and  products  needed  to  satisfy  the  terms  of  reference. 
Independent  and  intervening  variables  are  also  chosen 
based  on  the  purpose  of  the  analysis. 

In  the  initial  problem  formulation  iteration,  it  is  critical 
to  begin  with  an  understanding  of  the  REAL  problem 
rather  than  a  determination  to  apply  readily  available 
tools,  scenarios,  and  data. 
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Within  the  NATO  context,  a  number  of  documents  are 
available  or  under  development  that  may  assist  in 
understanding  the  study  context.  They  are  listed  at 
the  end  of  this  chapter. 

Principles  of  Problem  Formulation 

There  is  no  universally  acceptable  approach  to 
problem  formulation.  However,  best  practices  exist  that 
can  be  applied.  The  principles  associated  with  problem 
formulation  are  addressed  in  two  categories:  those  that 
are  appropriate  for  all  C2  assessments  and  those  that 
are  appropriate  for  assessments  of  C2  for  Operations 
Other  Than  War  (OOTW). 

Principles  Appropriate  for  C2  Assessments 

Explicit  problem  formulation  must  precede  construction 
of  concepts  for  analysis  or  method  selection.  This  is 
not  a  trivial  exercise,  especially  in  C2  assessments. 
Proper  resourcing  of  problem  formulation  activities  will 
improve  the  overall  efficiency  and  quality  of  the  study. 

An  understanding  of  the  decisions  to  be  supported  by 
the  analysis  and  the  viewpoints  of  the  various 
stakeholders  (e.g.,  customers,  users,  and  suppliers) 
is  essential  to  clarifying  the  study  issues.  This 
understanding  should  be  fed  back  to  the  stakeholders. 
A  careful  review  of  previous  and  current  work  must  be 
carried  out  as  a  valuable  source  of  ideas,  information, 
and  insight.  This  review  should  also  serve  to  identify 
pitfalls  and  analytic  challenges. 

Problem  formulation  must  not  only  provide  problem 
segments  amenable  to  analysis,  but  also  a  clear  and  valid 
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mechanism  for  meaningful  synthesis  to  provide  coherent 
knowledge  about  the  original,  larger  problem.  The 
formulated  problem  is  thus  an  abstraction  of  the  real  problem 
that  can  be  defined  in  terms  of  dependent  variables  that 
relate  to  this  real  problem  and  coherent  settings  for  the 
independent  variables  that  can  be  interpreted  in  terms  of 
decisions  and  actions  by  the  customer. 

Problem  formulation  must  be  broad  and  iterative  in 
nature,  accepting  the  minimum  of  a  priori  constraints 
and  using  methods  to  encourage  creative  and  multi¬ 
disciplinary  thinking,  such  as  proposing  a  number  of 
hypotheses  for  the  expression  of  the  problem.  It  must 
be  recognised  that  change  is  inevitable  in  many 
dimensions  (e.g.,  understanding  of  the  problem, 
requirements,  technologies,  co-evolution  of  concepts 
of  operation,  command  concepts,  organisation, 
doctrine,  systems).  Thus  the  assessment  process  must 
anticipate  and  accommodate  this  change. 

Practical  constraints  such  as  data  availability,  study 
resources  (including  time),  and  limitations  of  tools 
should  be  treated  as  modifiers  of  the  problem 
formulation  rather  than  initial  drivers.  Such  constraint 
may,  in  the  end,  drive  the  feasible  solutions,  but  it  is 
important  to  recognise  this  as  a  compromise  rather 
than  an  ideal.  Proper  problem  formulation  takes 
substantial  time  and  effort! 

It  is  important  that  problem  formulation  address  risk 
from  multiple  perspectives.  In  addition  to  sensitivity 
analysis  of  the  dependent  variables,  risk  analysis 
techniques  should  be  used  to  directly  explore  options 
to  mitigate  risk  (Chapter  10). 
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C2  assessment  often  involves  impacts  on  defense 
business  outside  the  context  of  a  particular  campaign 
or  operation.  The  study  must  address  these  impacts. 

Principles  Appropriate  for  OOTW  C2 
Assessments 

Problem  formulation  must  address  the  geopolitical 
context  of  the  problem  and  seek  to  identify  the  “broad” 
C2  issues  contained  within  the  terms  of  reference  for 
the  study.  There  are  no  universal  societal  “norms.” 
Therefore,  care  must  be  taken  in  attempting  to  transfer 
the  experience  in  one  OOTW  to  another. 

OOTW  C2  assessments  often  involve  policy-related 
impacts  outside  the  context  of  a  particular  military 
operation.  Therefore,  MoM  hierarchies  must  contain 
measures  of  policy  effectiveness. 

An  historical  perspective  is  critical  to  understanding 
OOTW  because  social  conflict  and  structures  often 
have  roots  far  back  in  history.  However,  it  must  be 
remembered  that  present-day  social  behaviour  is  not 
driven  by  historical  events  themselves,  but  by  present- 
day  perceptions,  processes,  and  prejudices  which 
have  evolved  from  the  past. 

A  key  risk  in  complex  OOTW  studies  is  allowing  the 
problem  formulation  process  to  focus  prematurely  on 
subsets  of  the  problem  because  they  are:  a)  interesting; 
b)  familiar;  c)  pre-judged  to  be  critical;  or  d)  explicitly 
called  out  by  the  customer.  This  requires  great  discipline 
by  the  study  team,  especially  where  the  team’s  previous 
experience  is  biased  in  favour  of  particular  parts  of  the 
problem  space.  The  assessment  team  needs  access 
to  subject  matter  experts  from  a  broad  range  of 
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disciplines  (e.g.,  social  scientists,  historians,  and 
regional  experts  in  0071/1/  assessment). 

Problem  Formulation  Process 

During  the  early  stages  of  problem  formulation  it  is 
important  to  quickly  cover  the  whole  problem  and 
produce  an  initial  formulation  (i.e.,  an  explicit 
expression  of  the  problem)  (See  Figure  3.1).  This 
prevents  premature  narrowing  of  the  assessment  and 
serves  as  an  aid  to  shared  situation  awareness  within 
the  study  team. 


Formulated  Problem 

bsues  to  b*  adtfmMd 
Asurrpooni 
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•  Unocnrolable 
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Figure  3.1.  The  Formulated  Problem 

The  process  begins  with  the  sponsor  presenting  the 
assessment  team  with  a  problem  to  assess  and  an 
articulation  of  broad  constraints  (e.g.,  schedule, 
resources).  Based  on  a  preliminary  assessment  of  the 
problem,  the  team  identifies  the  key  issues  to  address. 


60 


NATO  Code  of  Best  Practice  for  C2  Assessment 


This  identification  of  key  issues  leads  to  a 
characterisation  of  the  context  for  the  study  (e.g., 
relevant  geopolitical  factors,  identification  of  the  key 
actors  and  threats,  identification  of  generic  C2  issues, 
review  of  prior  studies).  Based  on  the  results  of  this 
characterisation,  the  analysis  team  identifies  what  it 
perceives  as  the  real  issues  to  address.  It  is  vital  for  the 
team  to  engage  in  a  dialogue  with  the  key  sponsor  and 
stakeholders  to  get  “buy  in”  for  these  issues.  Once  that 
is  achieved,  the  team  must  identify  and  characterise 
the  remaining  elements  of  the  problem  formulation 
phase.  To  facilitate  that  activity,  the  analysis  team  should 
identify/create  and  apply  selected  problem  formulation 
tools  and  techniques  (e.g.,  brainstorming,  Delphi 
analyses,  directed  graphics,  influence  diagrams).  The 
results  of  that  activity  will  include  a  summary  of  the 
assumptions,  high-level  MoM,  independent  variables 
(both  controllable  and  uncontrollable),  and  constraints 
on  the  variables.  Once  it  is  coordinated  with  the  sponsor 
and  stakeholders,  the  end  product  documents  what  is 
to  be  done  in  the  analysis.  The  next  key  activity  will  be 
to  develop  a  solution  strategy  that  describes  how  the 
study  is  to  be  done. 

Bounding  the  Problem/lssues  and 
Assumptions 

In  dealing  with  fuzzy  or  uncertain  boundaries,  the 
problem  formulation  process  needs  to  explore  and 
understand  the  significance  of  each  boundary  before 
making  (or  seeking  from  customers)  assumptions 
about  it.  This  involves  keeping  an  open  mind,  during 
the  early  stages  of  problem  formulation,  about  where 
the  boundaries  lie  and  their  dimensional  nature.  This 


Chapter  3  61 


is  difficult  because  it  makes  the  problem  modelling 
process  more  complicated.  A  call  for  hard  specification 
too  early  in  the  problem  formulation  process  must  be 
avoided.  In  the  end,  of  course,  the  problem  must  be 
formulated  in  order  to  solve  it,  but  formulation  should 
be  an  output  from  the  first  full  iteration,  not  an  early 
input  to  it. 

In  formulating  an  OOTW  problem,  we  are  trying  to 
bound  a  complex  system.  This  is  partly  a  process  of 
understanding  boundaries  which  exist  in  reality  (e.g., 
mission  statements,  geographical  areas  and  the  timing 
of  a  procurement  process)  and  partly  imposing  artificial 
boundaries  in  order  to  illuminate  the  structure  of  the 
problem  and  constrain  the  scope  of  the  analysis.  To 
avoid  the  trap  of  over-specification,  boundaries 
(especially  self-imposed  ones)  should  be  kept  porous, 
allowing  for  cause  and  effect  chains  to  flow  through 
the  external  environment  of  the  portion  of  the  complex 
system  that  the  boundaries  define. 

While  clear  definitions  and  hard  conceptual  boundaries 
are  ultimately  necessary  in  order  to  create  a 
manageable  problem  space,  care  must  be  taken  to 
avoid  coming  to  closure  prematurely. 

High-Level  MoM 

Identification  of  high-level  MoM  should  start  with  ideal 
measures  of  the  desired  benefits  before  considering 
what  can  be  practically  generated  by  analysis  (the  latter 
may  force  the  use  of  surrogate  MoM,  but  these  must 
be  clearly  related  to  the  desired  measures). 

A  structured  analysis  of  potential  benefits2  should  be 
carried  out  as  a  basis  for  constructing  appropriate 
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MoM.  Mapping  techniques,  such  as  cognitive  and 
causal  mapping,  are  a  good  way  to  express  the  various 
relationships  within  the  problem  space  and  to  identify 
‘chains’  of  analysis  (i.e.  links  among  the  independent 
variables  and  between  the  independent  and  dependent 
variables).  These  lead  to  resultant  structure  in  terms 
of  independent  and  dependent  variables,  and  hence 
to  high-level  MoM. 

Problem  Formulation  Tools 

It  is  useful  to  identify,  develop  (if  necessary),  and  apply 
appropriate  tools  to  support  problem  formulation. 
Representative  tools  and  techniques  include: 
techniques  for  supporting  expert  elicitation,  influence 
diagrams,  causal  maps,  system  dynamic  models,  and 
agent-based  models. 

Problem  Formulation  is  fundamentally  a  social  process 
of  developing  a  shared  understanding.  People  skills 
such  as  the  ability  to  facilitate  a  ‘brainstorming  session’ 
or  to  elicit  information  and  context,  are  thus  important. 
‘Throwaway  models’  (which  may  be  simple  simulation 
models,  causal  maps,  system  dynamic  models,  etc.) 
may  be  developed  as  part  of  the  process,  and  then 
discarded  as  insight  is  gained. 

Tools  and  approaches  used  for  problem  formulation 
must  be  consistent  with  other  tools  and  techniques 
likely  to  be  considered  for  the  subsequent  analysis  in 
order  to  produce  a  sensible  ‘multi-methodology’ 
approach  to  the  entire  problem  and  its  solution. 
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Constraints  on  the  Variables 

The  formulation  of  the  problem  is  completed  when  the 
constraints  on  either  the  independent  or  dependent 
variables  have  been  identified.  Constraints  on  the 
dependent  variables  represent  “acceptable”  thresholds 
or  limits.  For  example,  one  could  place  a  constraint 
on  blue  loss,  time  to  accomplish  a  mission,  collateral 
damage,  or  some  combination  of  factors.  Constraints 
on  the  independent  variables  represent  either  feasible 
or  acceptable  limits  on  such  factors  as  human 
performance,  C2  system  performance,  or  even 
supplies.  They  also  could  represent  doctrinal  or  legal 
processes  that  act  as  constraints. 

The  Next  Step 

The  next  step  in  the  C2  assessment  process  is  the 
development  of  a  solution  strategy.  It  should  be  noted 
that  the  team  is  not  finished  with  problem  formulation 
at  this  point  but  is  now  ready  to  proceed  to  build  a 
solution  strategy.  As  work  progress  on  the 
development  of  a  solution  strategy,  it  will  also  certainly 
be  necessary  to  revisit  the  specification  of  high-level 
MoM  and  the  constraints.  This  chapter  concludes  with 
a  discussion  of  the  products  of  problem  formulation. 

Products  of  Problem  Formulation 

Figure  3.2  depicts  the  essential  elements  of  the 
formulated  problem. 
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Figure  3.2.  Problem  Formulation 

A  checklist  can  be  used  to  ensure  that  all  the  aspects 
described  in  the  definition  have  been  covered.  These  include: 

•  Precise  statements  of  the  question  being  researched; 

•  A  list  of  independent  variables; 

•  A  list  of  high-level  MoM;  and 

•  A  list  of  assumptions  and  constraints. 

Diagrams 

Typically,  the  problem  formulation  phase  should  also 
produce  a  number  of  diagrams  such  as  influence  maps 
which  summarise  the  key  issues  and  interactions. 
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Data  Glossary 

The  problem  formulation  phase  must  begin  to  create 
a  glossary  of  key  data  elements,  metadata, 
information,  and  terms. 

Chapter  3  Acronyms 

C2  -  Command  and  Control 

MoM  -  Measures  of  Merit 

OOTW  -  Operations  Other  Than  War 

TTP- Tactics,  Techniques,  and  Procedures 
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'Broad  C2  issues  include  key  systems,  doctrine,  Tactics, 
Techniques,  and  Procedures  (TTP),  organisational  structures, 
and  key  assumptions  (e.g.,  system  performance  parameters). 
2The  structured  analysis  of  benefits  is  a  logical  process  that 
seeks  causally  to  map  lower  level  MoM  that  can  be  related  to 
investments  or  other  actions  to  higher  level  MoM  that  can  be 
valued  directly  by  decisionmakers. 
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Solution  Strategies 


Operations  research  is  a  scientific  method. 
Executives  have  often  in  the  past  used  some 
of  the  techniques... to  help  themselves  arrive 
at  decisions... But  the  term  “scientific  methods” 
implies  more  than  sporadic  application  and 
occasional  use  of  a  certain  methodology;  it 
implies  recognized  and  organized  activity 
amenable  to  application  to  a  variety  of 
problems  and  capable  of  being  taught. 

— Philip  M.  Morse  and  George  E  Kimball, 
Methods  of  Operations  Research 
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The  Study  Plan 

A  conscientious  effort  is  required  to  create  and  follow 
a  study  plan  that  guides  data  collection  and 
analyses  and  prepares  for  the  use  of  the  insights  and 
data  to  be  collected  to  contribute  to  a  solution  to  the 
problem  at  hand.  The  study  plan  consists  of  two  inter¬ 
related  parts — the  formulated  problem  {the  What)  and 
the  solution  strategy  ( the  How).  The  output  of  the  initial 
problem  formulation  provides  the  assessment  team 
with  an  operating  definition  of  what  needs  to  be  done. 
The  output  of  the  solution  strategy  phase  provides  the 
team  with  an  operating  definition  of  how  this  will  be 
accomplished.  As  the  project  unfolds,  there  will  usually 
be  a  significant  amount  of  iteration  that  modifies  both 
the  problem  formulation  and  the  solution  strategy. 
Without  a  study  plan,  it  is  unlikely  that  needed  efforts 
will  be  properly  scoped,  prioritised,  scheduled,  and 
resourced.  Even  if  the  way  ahead  seems  clear,  the 
articulation  of  a  formal  Solution  Strategy  is  necessary. 

The  objective  of  this  phase  of  the  study  is  to  develop 
a  feasible  approach  to  go  from  the  specification  of  what 
is  to  be  done  to  how  it  is  to  be  done.  This  involves 
developing  an  approach  that  will  result  in  the  team’s 
ability  to  collect  the  data  necessary  to  determine  the 
values  of  the  Measures  of  Merit  (MoM)  for  specified 
values  of  independent  variables.  The  characteristics 
of  data  collection  instruments  and  analysis  tools  and 
techniques  will  determine  the  resources  required,  the 
time  needed,  and  the  risks  inherent  in  the  solution 
approach.  When  compared  to  the  study  constraints 
and  the  problem  formulation,  it  will  be  determined 
whether  the  solution  approach  is  both  feasible  and 
satisfies  the  requirements  of  the  problem  formulation 
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(e.g.,  measures  the  right  things).  Figure  4.1  depicts 
what  is  involved  in  moving  from  a  problem  formulation 
to  a  solution  strategy. 


Figure  4.1 .  From  Problem  Formulation  to  Solution  Strategy 

Key  Definitions 

Solution  Strategy 

A  solution  strategy  consists  of  the  specification  of  a 
set  of  sequential  and  parallel  analytical  steps,  often 
involving  several  methodologies  and  tools.  The 
solution  strategy  is  designed  to  begin  with  what  is 
known,  and  by  execution  of  the  specified  steps,  leads 
to  what  one  desires  to  know — an  illumination  of  the 
issues.  The  strategy  can  be: 

•  simple — calculate  mortgage  payments  by  finding 
the  input  values  for  the  payment  equation  and 
then  evaluating  the  result; 
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•  moderately  complicated — define  input  variables, 
output  variables  and  precision  requirements, 
create  a  designed  experiment  and  run  the 
experiment  with  appropriate  measurements 
including  the  regression  analysis;  or 

•  extremely  complex — identify  the  relevant 
variables  and  systems  of  variables,  specify  how 
they  might  be  measured,  hypothesize  how  they 
are  related,  and  design  research  strategies  that 
allow  for  complex  adaptive  systems  or  other 
“messy”  structures  or  processes. 

The  solution  strategy  must  take  the  outputs  from 
problem  formulation,  refine  and  operationalise  them, 
and  develop  a  plan  to  collect  and  analyse  appropriate 
data  (including  the  development  and/or  selection  of 
models,  the  design  of  collection  instruments,  and  the 
selection  of  analysis  tools)  to  understand  the 
relationships  among  the  relevant  variables  associated 
with  MoM,  the  scenarios,  and  human  and 
organisational  factors.  (Figure  4.2) 

Measures  of  Merit  (MoM) 

MoM  are  a  set  of  variables  that  focus  the  assessment 
on  the  issues  of  interest.  In  most  analyses,  these  are 
the  dependent  variables.  In  many  cases  there  are 
significant  inter-relationships  among  the  MoM. 

Human  Factors 

Human  factors  consists  of  a  set  of  variables  that 
characterise  concepts  including  beliefs,  cultural  norms, 
stress,  fatigue,  fear,  arousal,  morale,  intelligence,  and 
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level  of  experience.  In  Command  and  Control  (C2) 
assessments  these  are  typically  independent  or 
intervening  variables. 

Organisational  Factors 

Organisational  factors  consist  of  a  set  of  variables  that 
characterise  organisations,  such  as  cohesion, 
command  structure,  explicit  and  tacit  relationships, 
information  flows,  and  organisational  cultures.  These 
are  also  typically  independent  of  intervening  variables 
in  C2  analyses. 

Scenarios 

Scenarios  consist  of  the  evolution  in  time  of  several 
elements:  a  context  (e.g.,  a  characterisation  of  a 
geopolitical  situation),  the  participants  (e.g.,  intentions, 
capabilities  of  blue,  red,  others),  and  the  environment 
(e.g.,  natural,  weather  and  manmade,  mines).  In  C2 
assessments,  the  purpose  of  scenarios  is  to  ensure 
that  the  analysis  is  performed  within  the  appropriate 
range  of  opportunities  to  observe  the  relevant  variables 
and  their  interrelationships. 

Model 

A  model  is  a  physical,  analogue,  or  symbolic 
representation  of  relevant  aspects  of  reality  for  a 
purpose.  It  is  an  abstraction  of  reality.  A  model 
emphasises  particular  aspects  (a  subset)  of  reality. 

The  assessment  model  of  primary  interest  is  the 
assessment  team’s  model  (conceptual  or  in  some 
analytic  manifestation)  of  the  C2  problem  including  the 


74 


NATO  Code  of  Best  Practice  for  C2  Assessment 


variables  of  interest,  their  hypothesised  relationships, 
and  any  prior  assumption  about  their  values  and 
linkages.  The  assessment  team  may  also  employ  or 
develop  other  models  or  simulations  in  order  to  perform 
analysis  or  explore  risks  and  uncertainties.  Some  teams 
will  employ  more  than  one  of  these  analytic  tools. 

Tool 

A  tool  facilitates  the  exploration  of  relationships  among 
model  variables  and/or  develops  “solutions”  (e.g., 
maximise  value  subject  to  constraints).  A  tool  may  be 
as  simple  as  a  checklist  or  an  algorithm,  or  it  may  be 
an  extremely  large  simulation.  A  simulation  is  the 
instantiation  of  a  model  that  serves  to  facilitate  the 
exploration  of  the  relationships  among  the  variables  - 
it  generates  data  for  analysis  and  generally 
emphasises  the  passage  of  time.  Models  and 
simulations  are  frequently  subdivided  into  categories 
of  constructive,  virtual,  and  live. 

Occasionally,  the  distinctions  among  a  tool,  model, 
and  data  are  subtle.  For  example,  in  a  linear  program 
the  model  is  the  set  of  formulas  that  specify  the 
objective  function  and  the  constraints.  The  tool  is  the 
simplex  method  (or  similar  algorithmic  solution 
method).  The  data  is  an  instantiation  of  the  formulas 
(provides  values  for  the  coefficients  and  constants). 
In  the  case  of  a  simulation,  the  simulation  environment 
and  simulation  engine  are  the  tools,  the  coded 
simulation  embodies  the  model,  and  the  input  values 
to  the  simulation  comprise  the  data.  Often,  the 
simulation  code  and  the  data,  together,  are  required 
for  a  complete  definition  of  the  model. 
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Data 

Data  are  the  values  associated  with  the  variables.  Data 
may  be  ratio,  interval,  ordinal,  or  nominal  in  scale.  Data 
may  originate  from  empirical  observation;  be  derived  from 
models,  simulations,  or  analyses;  be  established  from 
subject  matter  experts;  or  be  established  by  assumption. 

Developing  a  Solution  Strategy 

The  development  of  a  solution  strategy  is  an  iterative 
process  that  strikes  an  artful  balance  between  what 
the  team  would  like  to  do  and  what,  given  the  state  of 
the  art,  the  available  data,  tools,  schedule,  and 
resources,  is  possible  to  do. 

Prerequisites 

The  solution  strategy  should  not  be  designed  before 
the  problem  formulation  process  is  substantially 
complete  (Figure  4.1)  and  the  problem  formulation 
products  specified  in  Chapter  3  are  available  to  the 
team.  This  means  that: 

•  The  “real”  question  to  be  answered  is  known; 

•  The  assumptions  have  been  articulated; 

•  The  high  level  MoM  have  been  identified; 

•The  independent  variables  have  been  identified;  and 

•  The  constraints  associated  with  the  variables 
have  been  identified. 
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However,  the  assessment  team  should  always 
remember  the  inherently  iterative  nature  of  the 
process.  Adjustments  may  prove  necessary  in  the 
problem  formulation  as  the  solution  strategy  matures. 

Steps  in  Developing  a  Solution  Strategy 

As  an  initial  step,  the  team  should  elaborate  on  the 
MoM  to  specify  the  detailed  MoM  that  are  to  be 
evaluated.  This  is  sometimes  referred  to  as  developing 
operational  definitions  for  the  MoM — definitions  that 
specify  the  metric  to  be  used,  the  instrument,  and  the 
context  in  which  the  measurement  is  to  take  place. 
Often  the  value  of  a  particular  MoM  can  not  easily  be 
observed  or  measured  and  one  or  more  surrogate 
measures  are  used  in  its  place.  In  any  event,  the 
development  of  the  set  of  MoM  to  be  used  in  the  study 
anchors  the  process  that  will  eventually  lead  to  a 
solution  strategy. 

This  process  (Figure  4.2)  revolves  around  the 
conceptual  model  that  the  assessment  team  builds, 
and  is  at  the  heart  of  that  process.  It  is  best  practise  to 
make  this  model  explicit  and  have  it  serve  as  the 
common  picture  that  develops  a  high  quality  of  shared 
understanding  among  the  team,  sponsors, 
stakeholders,  and  other  key  study  participants.  The 
initial  conceptual  model  consists  simply  of  the  MoM,  a 
first  cut  of  the  hypothesised  relationships  among  them, 
assumptions  about  variables  and  their  relationships, 
and  constraints.  Later  iterations  include  additional 
independent  variables  that  are  known  or  assumed  to 
affect  the  values  of  the  MoM  or  the  nature  of  the 
relationship  among  them,  increasingly  detailed 
specifications  of  relationships,  and  specific  values  or 
ranges  for  the  independent  variables. 
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Figure  4.2.  Solution  Strategy 


The  identification  of  human  and  organisational  factors 
that  impact  model  variables  and  relationships  serves 
to  flesh  out  the  basic  conceptual  model  generated  in 
the  problem  formulation  phase. 

Scenarios  then  need  to  be  derived  to  provide 
opportunities  in  an  appropriate  context  for  data 
collection  and  exploration  of  the  variables  and 
relationships  contained  in  the  conceptual  model.  The 
data  the  study  requires  are,  in  large  measure,  a 
derivative  of  the  scenarios  utilized  and  the  design  of 
the  assessment. 

The  design  of  an  assessment  also  requires 
specification  of  methods  and  tools  and  how  they  will 
be  employed.  Methods  and  tools  are  required  to 
explore  the  relationships  among  the  independent 
variables  and  between  the  independent  and  dependent 
variables.  Complex  solution  strategies  may  be 
necessary.  In  these  cases,  multiple  analyses  will  be 
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implied.  The  problem  must  be  divided  into  parts,  each 
part  requiring  analysis  with  its  own  set  of  tools. 
Frequently  the  tools  that  are  available  do  not  provide 
interfaces  from  one  part  of  the  analysis  to  the  next. 

Taken  together,  the  detailed  specification  of  the  MoM, 
the  development  of  a  conceptual  model  including  the 
relevant  human  and  organisational  factors,  the 
specification  of  a  set  of  scenarios,  and  a  data  collection 
and  analysis  plan  (that  consists  of  the  methods  and 
tools  to  be  used)  constitute  a  solution  strategy. 

The  solution  strategy  developed  needs  to  be  tested  to 
see  if  it  can  be  expected  to  address  the  issues  at  hand, 
within  schedule  and  resource  constraints,  and  with 
accepted  levels  of  uncertainty  and  risk.  However, 
uncertainty  and  risks  are  being  continually  assessed 
throughout  the  process  of  developing  a  solution 
strategy.  The  team  should  also  consider  the  form  of 
study  output  and  its  relevance  to  the  decisionmaker. 
Iteration  of  these  ideas  with  the  stakeholders 
throughout  the  study  helps  to  avoid  surprises  and  to 
ensure  that  the  basic  assumptions  underlying  the  study 
have  not  changed. 

Iterating  the  Study  Plan 

Figure  4.3  depicts  the  iterative  nature  of  the  process 
involved  in  developing  the  Overall  Study  Plan,  linking 
problem  formulation  and  solution  strategies  together 
with  the  inputs  from  study  sponsors  and  stakeholders. 
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Figure  4.3.  Overall  Study  Plan 


A  first  order  feedback  loop  is  shown  between  problem 
formulation  and  solution  strategy,  with  both  processes 
having  iterative  internal  processes.  An  analysis  of  the 
study  of  risk  and  uncertainty  provides  the  control 
mechanism  that  drives  the  iterative  process  to  an 
acceptable  result. 

Study  Management  Plan 

The  team  should  also  create  and  maintain  a  Study 
Management  Plan  (SMP)  to  guide  the  direction, 
management  and  co-ordination  of  the  project  team.  The 
SMP  should  include  a  detailed,  time-phased  execution 
plan  for  the  study  and  a  Work  Breakdown  Structure 
(WBS).1  The  SMP  should  show  the  requirements  for  all 
the  team  products  and  their  delivery  dates  thereby 
creating  delivery  milestones  for  the  execution  of  the 
study.  It  should  show  the  planned  dates  for  all  scheduled 
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meetings  including  progress  meetings  and  technical 
interchange  meetings.  The  plan  should  also  include  a 
time-phased  manning  plan  identifying  the  types, 
quantities,  and  period  of  performance  for  all  members 
of  the  study  team.  The  SMP  should  include  details  of 
the  controls  that  will  be  applied  to  supervise  any 
contractor  performance.  The  team  should  maintain  a 
current  version  of  the  SMP  during  the  study  period  of 
performance.  The  SMP  would  typically  include  the 
associated  supporting  plans: 

•  Study  glossary; 

•  Analysis  plan; 

•  Tool  deployment  and  modelling  and  simulation  plan; 

•  Data  collection/engineering  plan; 

•  Configuration  management  plan; 

•  Explicit  consideration  of  risk  and  uncertainty; 

•  Quality  assurance  plan; 

•  Security  plan; 

•  Review  plan;  and 

•  Plan  of  deliverables. 

The  elements  of  an  ideal  SMP  are  defined  and 
discussed  below. 

Study  Glossary 

The  assessment  team  should  create  and  maintain  a 
study  glossary  comprising  all  relevant  definitions 
needed  in  the  study.  It  should  aim  to  create  a  general 
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study  glossary  that  is  improved  by  every  study  that 
uses  this  glossary.  As  a  starting  point,  the  NATO  AAP- 
6,  “NATO  Glossary  of  Terms  and  Definitions,”  or  the 
Joint  Publication  1  -02,  “DOD  Dictionary  of  Military  and 
Associated  Terms,”  should  be  used. 

Analysis  Plan 

The  assessment  team  should  also  create  and  maintain 
an  analysis  plan  for  the  study.  The  analysis  plan  should 
describe  the  analyses  in  detail.  This  description  should 
include  the  analysis  methodology,  the  tools  to  be  used 
for  analysis,  the  input  data  requirements,  the  Essential 
Elements  of  Analysis  (EEA),  the  MoM  to  be  used  to 
evaluate  the  results,  and  any  analysis  assumptions. 

Tool  Deployment  and  Modelling  and 
Simulation  Plan 

The  team  may  create  and  maintain  a  tool  deployment 
and  modelling  and  simulation  plan  covering  the  needs 
of  each  task  for  numerical  simulations  or  other  applied 
means  of  operations  analysis  (OA).  It  should  describe 
the  use  of  tools  and  models  and  simulations  in  the 
feasibility  study  (FS).  The  plan  should  include  a 
description  of  each  tool  to  be  used,  a  list  of  the  key 
assumptions  and  caveats  for  each  tool,  an  analysis  of 
the  suitability  of  each  tool  in  addressing  the  functionality 
and  performance  issues  in  the  FS,  the  source  of  input 
data  for  each  tool,  the  available  output  from  each  tool, 
and  should  detail  any  changes  to  these  tools  that  are 
intended.  The  plan  should  indicate  how  data 
traceability  from  one  tool  to  another  will  be  maintained. 
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The  modelling  and  simulation  plan  may  be  included 
as  part  of  the  analysis  plan  in  simple  studies. 

Data  Collection/Engineering  Plan 

The  team  should  create  and  maintain  a  Data 
Collection/Engineering  Plan  (DCEP)  which  covers  data 
and  metadata  necessary  to  describe: 

•  The  scenario; 

•  The  essential  elements  of  analysis  in  the  analysis  plan; 

•  The  MoM  to  be  used  to  evaluate  the  result  (also 
in  the  analysis  plan);  and 

•  The  input  and  output  parameters  of  tools  to  be 
used  within  the  study. 

The  DCEP  describes  and  documents  who  owns  the 
data,  where  the  data  can  be  found  (including  open 
sources  like  the  Web),  necessary  methodologies  and 
procedures  to  prepare  the  data,  and  assumptions  and 
constraints  connected  to  generated  data,  etc.  The  data 
definitions  used  in  the  DCEP  have  to  be  harmonised 
with  the  study  glossary. 

Configuration  Management  Plan 

The  team  should  create  and  maintain  a  configuration 
management  plan.  The  plan  should  ensure 
identification,  traceability  and  control  of  the  descriptions 
of  the  system  elements,  interfaces  and  architectures 
considered  in  the  FS,  as  well  as  associated 
documentation.  The  plan  should  show  how  the  study 
database  must  be  controlled  and  updated.  The  plan 
should  follow  NATO  STANAG  4159,  “NATO  Materiel 
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Configuration  Management  Policy  and  Procedures  for 
Multi-National  Joint  Projects”  (1992),  as  a  guideline. 
The  plan  should  also  ensure  that  the  description  of 
the  system  configuration  being  simulated  or  analysed 
can  be  precisely  identified,  as  well  as  any  system  or 
technology  improvements  considered,  with  respect  to 
an  identified  baseline  configuration. 

Study  Risk  Register 

The  team  should  also  identify  and  assess  the  technical 
and  schedule  risks  concerned  with  the  successful 
completion  of  the  FS.  The  list  of  risks  identified  should 
be  maintained  in  a  study  risk  register,  which  shows 
the  probability  of  occurrence  of  each  risk  and  its  impact 
on  the  FS.  The  register  should  include  the  risk 
mitigation  activity  for  each  risk  and  the  expected 
improvements  to  time  and  performance.  The  study  risk 
register  should  be  regularly  maintained,  should  be 
available  to  the  assessment  team  by  arrangement,  and 
should  be  presented  at  each  progress  meeting.  The 
Generic  Risk  Register  (GRR)  developed  during  the 
SAS-026  efforts  (Chapter  10)  is  recommended. 

Quality  Assurance  Plan 

The  team  should  create  and  maintain  a  quality 
assurance  plan.  The  plan  should  declare  all  relevant 
quality  standards  and  procedures  that  are  to  be  applied 
in  the  course  of  the  study,  and  should  describe  the 
quality  organisation  to  be  used,  including  the  principal 
quality  officers  and  their  lines  of  authority.  The  policy 
and  requirements  for  quality  assurance  in  NATO  are 
given  in  the  following  documents,  which  should  be  used 
as  guidelines: 
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•  STANAG  4107,  “Mutual  Acceptance  of 
Government  Quality  Assurance  and  Usage  of 
the  Allied  Quality  Assurance  Publications 
(AQAPs),”  6th  Edition  (March  1998). 

•  AQAP  1 00,  “General  Guidance  on  NATO  Quality 
Assurance,”  2nd  Edition  (March  1995)  or 
equivalent/comparable  national  standards. 

The  Security  Plan 

The  team  should  create  and  maintain  a  security  plan. 
This  plan  should  contain  the  approach  to  the  utilisation, 
storage,  publication,  dissemination,  and  control  of 
classified  and  unclassified  materials. 

Review  Plan 

The  team  should  create  and  maintain  a  review  plan. 
For  every  critical  phase  of  the  study,  preferably  marked 
by  respective  milestones,  reviews  of  the  study  have 
to  be  planned  and  executed.  Participants  should  go 
beyond  the  members  of  the  study  team  to  include  peer 
reviews.  The  review  results  should  be  documented. 

Plan  of  Deliverables 

The  team  should  also  create  and  maintain  a  plan  of 
deliverables  for  each  phase  of  the  study.  This  includes 
what  is  to  be  delivered,  when  it  is  to  be  delivered,  to 
whom,  in  what  form  and  format,  and  how  many  copies. 

Chapter  4  Acronyms 


C2  -  Command  and  Control 
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DCEP  -  Data  Collection/Engineering  Plan 

EEA-  Essential  Elements  of  Analysis 

FS-  Feasibility  Study 

GRR  -  Generic  Risk  Register 

MoM  -  Measures  of  Merit 

OA  -  Operations  Analysis 

SMP  -  Study  Management  Plan 

WBS  -  Work  Breakdown  Structure 
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'A  WBS  is  a  decomposition  of  the  effort  into  its  constituent  parts 
(or  tasks)  and  the  assignment  of  assets  to  the  component  tasks. 
Assets  may  be  people’s  time,  facilities,  or  other  elements 
required  to  complete  the  task.  WBS  are  common  in  construction 
and  engineering  projects. 


CHAPTER  5 


Measures  of  Merit 


It’s  best  to  know  what  you  are  looking 
for,  before  you  look  for  it. 

— Winnie  the  Pooh,  from  A.  A.  Milne 


In  order  to  understand  the  impact  of  Command  and 
Control  (C2),  it  is  necessary  not  only  to  analyse  and 
measure  the  effect  of  C2  on  military  operations,  but 
also  the  effects  on  the  components  of  the  constituent 
systems.  No  single  measure  or  methodology  exists 
that  satisfactorily  assesses  the  overall  effectiveness 
of  C2.  Therefore,  a  multi-faceted  and  sometimes  multi- 
phased  approach  is  necessary.  The  benefits  of  C2 
should  be  evaluated  through  their  impact  on  the 
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fulfilment  of  the  military  and  policy  objectives,  and  the 
impact  of  C2  should  be  measured  in  terms  of  specific 
qualities  that  are  relevant  to  these  objectives.  A  set  of 
scenarios  provides  the  contexts  in  which  Measures  of 
Merit  (MoM)  are  determined. 

MoM  Challenges 

During  the  last  two  decades,  many  new  automated 
C2  systems  have  been  developed  and  fielded. 
However,  the  determination  of  both  the  performance 
and  the  effectiveness  of  these  systems  has  proven  to 
be  a  complex  problem.  Recognising  this,  the  Military 
Operations  Research  Society  (MORS)  has  sponsored 
several  workshops  on  MoM  since  1985.  The 
workshops  have  led  to  the  development  of  an  analysis 
framework,  Modular  Command  and  Control  Evaluation 
Structure  (MCES),  for  the  measurement  of 
performance  and  effectiveness  within  a  conceptual 
model  for  C2.  Based  on  the  MORS  workshops,  the 
US  Army’s  Training  &  Doctrine  Command  Analysis 
Center  (TRAC)  developed  the  C2  Measures  of 
Effectiveness  (MoE)  Handbook  in  1990.  This 
document  and  the  measurement  tools  developed  for 
the  Headquarters  Effectiveness  Assessment  System 
(HEAT)  and  the  Army  Command  and  Control 
Evaluation  System  (ACCES)  represented  the  then 
established  best  practices. 

The  AC/243  Panel  7  Ad  Hoc  Working  Group  (AHWG) 
on  the  Impact  of  Command,  Control,  Communications 
&  Intelligence  (C3I)  on  the  Battlefield  acknowledged  that 
the  specification  of  measures  of  effectiveness  is  difficult. 
The  1 992  final  report  recommended  that  a  hierarchy  of 
measures  be  established  as  an  important  step  in 
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understanding  overall  system  effectiveness,  and  that 
systems  be  analysed  at  different  levels  of  detail.  The 
types  of  measures  were  grouped  relating  to  C2  system 
performance,  force/commander  effectiveness,  and 
battle  outcome.  To  quote  from  the  final  report, 
“Measures... are  often  inadequate  and  too  model  or 
scenario  specific.  In  addition,  they  have  often  been 
generated  in  ad  hoc  ways,  suggesting  a  lack  of  formal 
analysis  in  their  development.”  Since  then,  RSG-19, 
SAS-002  and  SAS-026  have  canvassed  the  field,  and 
have  brought  together  the  best  ideas  and  practices  in 
order  to  support  MoM  development  and  applications 
within  C2  assessment.1  This  version  of  the  Code  of  Best 
Practise  (COBP)  extends  this  thinking  and  includes  the 
Operations  Other  Than  War  (OOTW)  domain. 

Definitions 

It  has  been  recognised  that  a  single  definition  for 
measures  of  performance  (MoP)  and  effectiveness 
(MoE)  does  not  exist.  MoM  is  recommended  as  a 
generic  term  to  encompass  different  classes  of 
measures.  The  measures  are  defined  in  hierarchical 
levels  related  to  each  other,  each  in  terms  of  its  own 
boundary.  From  the  conceptual  viewpoint,  it  is 
important  to  keep  in  mind  the  level  of  analysis  and  the 
context  in  which  the  measurements  are  made. 

Within  the  MCES  framework,  MORS  has  developed  a 
four-level  hierarchy  of  measures  from  high-level  force 
effectiveness  to  low-level  rudimentary  measures  of 
physical  entities,  which  were  adopted  by  RSG-19.  In 
the  context  of  OOTW,  a  fifth  level  is  added,  Measures 
of  Policy  Effectiveness  (MoPE),  to  characterise  the 
contribution  of  military  actions  to  broader  policy  and 
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societal  outcomes.  For  OOTW,  political  factors  are 
paramount  and  considerations  such  as  media 
coverage,  local  regional  stability,  and  sustainment  of 
community  societal  standards  must  be  taken  into 
account.  Military  missions  may  not  directly  achieve 
policy  objectives,  although  they  often  strive  to  provide 
an  environment  more  conducive  to  these  objectives. 
However,  MoE  of  military  tasks  should  quantify 
performance  against  military  missions,  not  the  overall 
political  aspirations. 

The  Code  of  Best  Practice  has  adopted  the  following 
five  levels  of  MoM: 

•  Measures  of  Policy  Effectiveness  (MoPE),  which 
focus  on  policy  and  societal  outcomes; 

•  Measures  of  Force  Effectiveness  (MoFE),  which 
focus  on  how  a  force  performs  its  mission  or  the 
degree  to  which  it  meets  its  objectives; 

•  Measures  of  C2  Effectiveness  (MoCE),  which 
focus  on  the  impact  of  C2  systems  within  the 
operational  context; 

•  Measures  of  Performance  (MoP),  which  focus  on 
internal  system  structure,  characteristics  and 
behaviour;  and 

•  Dimensional  Parameters  (DP),  which  focus  on 
the  properties  or  characteristics  inherent  in  the 
physical  C2  systems. 

Figure  5.1  emphasises  the  diminishing  impact  of  a 
particular  MoM  as  the  circle  widens. 
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Environment 


Figure  5.1 .  Relationships  of  Measures  of  Merit 

Measurement  Objectives 

The  important  issues  raised  by  decision-makers 
require  a  sense  of  the  degree  to  which  C2  performance 
may  improve  force  and  policy  effectiveness.  Therefore, 
C2  assessments  are  often  called  upon  to  provide 
convincing  evidence  of  the  expected  improvements 
in  mission  effectiveness  that  can  be  attributed  to 
improved  C2.  The  ideal  approach  may  be  to  define  a 
single  measure  that  reflects  the  military  and  political 
objectives  of  the  missions  under  consideration. 
However,  the  determination  of  such  a  measure  is 
generally  not  feasible,  although  not  necessarily 
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impossible  for  particular  classes.  In  most 
decisionmaking  problems,  it  is  necessary  to  define 
several  measures  that  together  provide  the  necessary 
insights.  A  major  reason  for  this  is  that  a  single 
measure  may  not  provide  sufficient  scope  and/or  detail 
to  analyse  the  impact  of  specific  C2  elements, 
particularly  second  and  third  order  effects  or 
unintended  consequences.  Many  analyses  are 
conducted  precisely  in  order  to  enable  trade-off 
between  important  equities  which  can  only  be  seen  if 
a  set  of  MoM  is  generated  for  analysis.  The  set  of 
MoM  selected  must  be  comprehensive  to  ensure  that 
all  factors  are  considered. 

MoM  are  used  to  compare  different  options  on  equal 
terms,  and  serve  a  wide  range  of  purposes,  including: 

•  Establishing  a  standard  or  expectation  of 
performance  (for  new  requirements); 

•  Establishing  the  bounds  of  performance  of  a  system 
as  well  as  the  effects  of  imposed  constraints; 

•  Comparing  and  selecting  alternative  systems  that 
may  be  very  dissimilar  but  are  designed  to 
achieve  a  similar  purpose; 

•  Assessing  the  utilisation  of  a  system  in  new  or 
unexpected  application  domains  or  missions; 

•  Identifying  potential  weaknesses  in  specific  areas 
of  an  organisation  or  system  (areas  of  high  error 
potential  or  high  user  workload); 

•  Analysing  the  impacts  of  organisational  changes; 

•  Analysing  training  effectiveness; 
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•  Determining  the  most  cost-effective  approaches 
to  achieve  desired  objectives; 

•  Comparing  a  replacement  system,  or 
components  of  a  system,  against  predecessors 
or  competitors; 

•  Assisting  in  generating  and  validating 
requirements  and  deriving  specific  C2 
requirements  from  broad  statements  of  objectives; 

•  Evaluating  the  effectiveness  of  human 
decisionmaking  in  the  C2  cycle; 

•  Determining  the  degree  of  mission  success;  and 

•  Determining  the  return  to  normality  in  OOTW. 

Characteristics  of  MoM 

The  development  of  an  operational  definition  of  the 
measure,  the  development  of  instruments,  the 
application  of  the  instruments  to  collect  of  appropriate 
data,  and  the  establishment  of  relationships  among  a 
group  of  MoM  vary  in  difficulty,  effort,  cost,  precision, 
generalizability  and  other  characteristics.  These 
differences  are  often  related  to  the  type(s)  of  MoM 
involved  and  the  domain  in  which  the  measurement 
needs  to  take  place. 

C2  Assessments  involve  measurement  of  variables 
that  exist  in  the  physical,  information,  and  cognitive 
domains.2  In  general,  the  development  of  operational 
definitions,  instrumentation  and  data  collection  for 
variables  to  be  measured  in  the  physical  and 
information  domains  are  more  straightforward  and 
require  less  effort  and  expense  than  dealing  with 
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variables  that  are  measured  in  the  cognitive  domain. 
Furthermore,  the  former  can  be  measured  more 
“precisely,”  are  easier  to  comprehend,  and  are  less 
subject  to  interpretation  than  the  latter. 

In  general,  DP,  MoP  related  to  systems,  and  MoFE 
tend  to  be  measured  in  the  physical  domain  (e.g., 
bandwidth,  computing  capacities,  time  to  accomplish 
a  task,  force  exchange  ratios)  while  MoP  related  to 
measures  of  C2  effectiveness  (quality  of  awareness, 
shared  awareness,  and  trust)  and  MoPF  (will  of  an 
adversary,  public  opinion)  tend  to  be  measured  in  the 
cognitive  domain. 

As  one  goes  up  the  hierarchy  of  MoM,  the  measures 
tend  to  become  more  context,  task,  or  mission  specific. 
For  example,  the  performance  characteristics  of  systems 
(DP)  apply  to  systems  in  general  but  MoFE  are  usually 
limited  to  a  set  of  tasks  or  missions.  MoFE  for  combat 
are  very  different  from  MoFE  for  various  OOTW.  If  done 
well  measures  of  C2  effectiveness  will  be  scenario 
independent  so  one  can  compare  C2  effectiveness 
across  a  range  of  missions  and  circumstances. 

Except  for  DP  any  of  the  MoM  can  be  either  an 
independent  or  dependant  variable  in  a  given 
assessment  with  any  of  the  independent  variables 
being  either  “controllable”  or  not.  The  difficulty  in 
establishing  relationships  among  the  MoM  varies  as 
a  function  of  the  level  of  the  independent  variable. 

The  Assessment  Team  should  recognize  and  plan  for 
the  difficulties  associated  with  using  various  MoM  and 
should  avoid  substituting  easier  to  deal  with  but  less 
relevant  MoM.  It  is  always  better  to  try  to  measure 
(estimate,  approximate)  MoM  that  reflect  first  order 
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effects  than  to  precisely  measure  MoM  that  do  not 
adequately  reflect  key  aspects  of  the  problem. 

Measuring  MoM 

This  section  outlines  measurement  theory  concepts 
that  apply  to  ensure  that  the  right  measuring 
instruments  are  selected  and  applied  correctly.  By 
definition,  measurement  is  the  assignment  of  values 
to  observation  units  that  express  properties  of  the  units. 
Four  levels  of  measures  relate  numbers  to  properties 
of  interest:  nominal  (e.g.,  artillery  vs.  infantry),  rank  or 
ordinal  (e.g.,  worst  to  best),  relative  or  interval  value 
(e.g.,  change  in  temperature),  and  absolute  value  or 
ratio  (e.g.,  2  kilobits  per  second).  Analysts  must  ensure 
that  the  specific  MoM  adopted  are  at  the  appropriate 
levels  of  measurement. 

The  key  properties  for  quality  assurance  are  reliability 
and  validity.  Other  significant  properties  include 
practical  issues,  such  as  the  effort  required  to  collect 
appropriate  data  and  the  convenience  of  measurement 
(e.g.,  whether  the  collection  process  itself  interferes 
with  the  conduct  of  an  exercise  or  experiment).  Ideally, 
measurements  should  be  easy  to  capture  and  easy  to 
apply.  There  are  clearly  trade-offs  to  be  made  between 
MoM  that  may  closely  track  the  property  of  interest 
and  that  are  costly  and/or  difficult  to  measure  and  those 
that  are  less  strongly  related  to  the  property  of  interest 
but  that  are  easier  to  measure.  The  effort  required  for 
collection  bears  no  direct  relationship  with  validity,  but 
reliability  may  be  related  to  cost.  Reliable 
measurements  require  repeated  observations  and 
appropriate  sample  sizes.  Reliability  represents 
accuracy  and  consistency.  A  cost-effective 
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measurement  plan  provides  enough  data  for  useful 
and  definitive  conclusions.  However,  cost  and/or 
convenience  of  measurement  may  be  an  overriding 
factor  in  system  evaluation. 

Failure  to  take  validity  and  reliability  into  account  raises 
the  risk  of  generating  false  conclusions.  Validity  and 
reliability  are  not  absolutes,  but  matters  of  degree. 
Validity  is  the  degree  to  which  a  measure  characterises 
the  attribute  of  interest  and  only  that  attribute.  Complex 
concepts  often  require  multiple  measures  to  provide 
valid  information.  In  order  to  make  a  valid  link  between 
the  performance  of  a  system  as  a  whole  against 
performance  of  its  components,  the  measures  must 
correspond  to  critical  tasks.  Reliability  represents 
accuracy  and  repeatability.  A  measure  may  be  reliable 
but  not  valid,  or  it  may  be  valid  but  not  reliable. 

Validity 

The  properties  of  validity  may  be  categorised  into  five 
types:  internal,  construct,  statistical,  external,  and  expert. 

•  Internal  validity  is  defined  as  the  establishment  of 
causal  relationships  between  variables  of 
interest.  This  is  necessary  to  accept  a 
hypothesis  that  a  given  measure  is  responsible 
for  a  specific  effect  on  another  measure; 

•  Construct  (also  referred  to  as  content)  validity 
means  that  the  target  objects,  and  only  the  target 
objects,  are  measured; 

•  Statistical  validity  implies  that  sufficient  sensitivity 
is  involved  in  order  to  determine  relationships 
between  independent  and  dependent  variables. 
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Statistical  tests  control  two  types  of  errors  in 
measurement.  Type  I,  or  alpha,  is  the  probability 
of  rejecting  a  claimed  hypothesis  that  is  true, 
Type  II,  or  beta,  is  the  probability  of  accepting  an 
hypothesis  that  is  not  true; 

•  External  validity  implies  that  the  results  may  be 
extended  to  other  populations  or  environments;  and 

•  Expert  validity  refers  to  the  degree  to  which 
measures  are  accepted  by  those  knowledgeable 
in  the  field. 


A  MoM  should  meet  the  validity-related  criteria  outlined 
in  Table  5.1 . 


Validity 

Criterion 

IMMMi 

Mission  O bunted 

Realistic 

Appropriate 

Inclusive 

Oecrmlnalory 

Meaningful 

Simple 

Relevant 

GeneralisaWe 

Relate*  to  foroaisystem  mtsvcn 

Relates  reafetcaiy  to  the  C2  system  and  esscoaled  uncertainties 

Relates  to  acceptable  standards  and  analyse  obtectves 

Reflects  these  standards  required  by  the  analysis  objectives 

Atte  to  dsttngueh  measurement  parameters 

Accepted  t >y  subject  metier  experts 

Easily  understood  by  users 

Target  objects.  and  only  the  target  objects  are  measured 

Extent  to  whkn  rebuts  may  be  extended  to  other  populations  or  envroomervs 

Table  5.1 .  Validity  Criteria  of  Measures 


Reliability 

Reliability  involves  the  expectation  of  errors  associated 
with  measurements.  It  is  defined  as  the  accuracy  of  a 
measurement,  as  reflected  in  the  variance  of  repeated 
measurements  of  the  same  phenomenon.  The  key 
principles  of  reliability  are  consistency  (repeatability) 
and  accuracy.  The  variance  associated  with 
measurement  must  be  known  or  estimated  to  interpret 
results  and  to  discriminate  between  real  effects  and 
measurement  effects. 
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The  reliability-related  criteria  that  MoM  should  meet 
are  outlined  in  Table  5.2.  Additional  criteria  that  have 
proven  usual  in  the  past  are  outlined  in  Table  5.3. 


ReliatMiry  Criterion 

Definition 

DecrniiPetory 

Vww  aote/ObwfYabtu 
Quantitative 

Object  v* 

Senatwe 

Const  91  eel 

Identbos  real  dirterences  bataeen  alternatrves 

Available  for  measurement 

Can  be  assigned  values,  numbers  or  ranks 

Defined  or  derived  cdepecde-il  of  subjective  opinion 

Abte  to  ascertain  changes  in  system  variables 

Same  results  ere  obtained  on  repatliion 

Table  5.2.  Reliability  Criteria  of  Measures 


Criterion  j 

DlllnIBbfi  1 

T  ri dries 5 
Adaptability 

Flexibility 

Robustness 

Sampling  rale  adequate  to  delect  changes 

Sufficient  within  available  resources 

Capability  across  a  range  of  anticipated  situations 

Capability  across  a  range  of  unanticipated  stuahors 

Table  5.3.  Other  Useful  Criteria 


Practical  MoM  Issues 

The  assessment  of  C2  requires  the  application  of  a 
framework  to  yield  values  for  appropriate  MoM. 
Analyses  of  C2  systems  and  processes  often  reveal  a 
complex  hierarchical  composition.  A  structured 
resolution/functional  decomposition  approach  may  be 
related  to  the  organisational  structure  to  yield 
performance  measures  for  the  organisation  as  a  whole, 
individual  components  within  the  organisation,  and 
specific  tasks  within  the  organisational  cells. 

If  the  analyst  assumes  that  C2  effectiveness  is  positively 
correlated  with  overall  military  unit  effectiveness,  MoM 
could  be  obtained  by  addressing  the  outcomes  or 
products  of  such  unit  activities.  Goal-level  evaluation 
attempts  to  define  the  ability  of  the  specific  military 
formation  to  make  the  system  state  match  the  goal 
(directive)  provided  by  the  superior  headquarters.  These 
are  measures  of  force  effectiveness.  The  degree  to 
which  the  system  state  matches  the  desired  goal  states 
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indicates  a  level  of  effectiveness.  Alternatively,  C2 
effectiveness  may  be  viewed  as  dependent  on  the 
functional  processes  of  the  C2  system,  with  measures 
obtained  mainly  at  the  task  level. 

A  C2  assessment  framework  encompasses  several 
factors  that  must  be  considered  iteratively,  as 
discussed  in  the  introductory  chapter.  Typical  factors 
important  for  the  identification  and  selection  of  MoM 
include  the: 

•  Assessment  configuration,  e.g.,  storyboard, 
testbed,  constructive  simulation,  field  trial; 

•  Assessment  goal  or  purpose; 

•  Context,  assumptions,  and  constraints; 

•  Scenarios  or  stimuli; 

•  Collection  means,  e.g.,  subject  matter  experts, 
automatic  data  logging; 

•  Analysis  plan;  and 

•  Interpretation  of  results. 

Categories  of  Measures 

A  common  thread  in  the  approaches  for  C2 
assessment  is  the  functional  decomposition  of  the  C2 
cycle.  C2  effectiveness  depends  upon  the  functional 
processes  of  the  C2  system,  and  the  evaluation  of 
functions  may  be  determined  by  data  measured  at  the 
task  level. 

The  evaluation  of  tasks  provides  the  most  detailed 
insight  into  C2  activities.  The  primary  measures  are 
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expressed  in  terms  of  time  consumed  and  accuracy. 
Task  analysis  must  be  performed  prior  to  evaluation, 
with  the  identification  of  task  definition  and  the  critical 
elements  for  successful  task  completion. 

Measures  of  a  C2  system’s  behaviour  may  thus  be 
reduced  to  measures  based  on  time,  accuracy,  or  a 
combination  which  may  be  interdependent.  Time 
based  measures  are  quantitative,  while  accuracy 
measures  may  be  quantitative  or  qualitative. 

For  C2  tasks,  time-based  metrics  include  the: 

•  Time  taken  to  react  to  an  event  (time  to  notice 
process  and  act  upon  new  information); 

•  Time  to  perform  a  task  (time  to  make  decision); 

•  Time  horizon  for  future  for  predictive  analysis;  and 

•  Rate  of  performing  tasks  (tempo). 

Metrics  for  accuracy  include: 

•  Precision  of  the  observed  system(s)  performance; 

•  Reliability  of  the  observed  system(s)  performance; 

•  Completeness  (known  unknowns,  unknown  unknowns); 

•  Errors  (alpha,  beta,  omission,  transposition, 
severity);  and 

•  Quality  of  information  produced. 

Some  accuracy  measures  may  be  calculated  in  units 
of  time,  e.g.,  the  time  taken  to  detect  an  error.  Quality 
of  decisions  is  difficult  to  evaluate  objectively,  except 
by  focusing  on  outcomes.  The  processes  involved  may 
have  to  be  examined  to  obtain  objective  measures,  or 
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subject  matter  experts  may  be  consulted  to  make  an 
evaluation.  Accuracy  of  information  implies  both  the 
accuracy  of  the  data  and  the  accuracy  of  the 
interpretation  of  the  data. 

Time  based  and  accuracy  measures  often  bear  an 
inverse  relationship,  implying  a  trade-off  between  speed 
of  performance  and  accuracy  of  performance.  Speed 
of  performance  must  be  specified  in  terms  of  minimum 
desired  accuracy  or  completeness,  and  accuracy 
measurements  in  terms  of  time  available.  Therefore, 
the  specification  of  thresholds  or  standards  for  metrics 
must  be  referenced  in  terms  of  imposed  constraints. 


Examples  of  time  and  accuracy  based  measures  are 
compiled  in  Table  5.4.  Table  5.5  provides  some 
additional  examples,  specifically  MoP  and  MoCE. 


Measure 

Example 

Tlnw  Ba—d 

T 10  fX*dorm  Of  ct  M*** 

PU»Y*f>9  'JAk-i, 

1  r-«  to  po'tomn  a  .anaoo  task 

Ooocprtc  and  sciortr^  cpticrs  «•  ccurco?  ot  zzt>y 

T r^f  to  reO0flT»«  Of  rrspro  to  an  e»ent 

to  *  CAk-3  en-r^y  <CCt»Ct 

Tr-«  to  ULi-ttr.u  a  uUitw 

TjKltCfll  C^twLlr.u 

^ercenta^e  c#  tr>*  cn  to'aet 

L\ya  bases  x»  to  date 

1 7  V.1  T I  y  V  •'  V  ■ 

'fTTf  V-  '-T-.'l  jp PI 

Tr  c  to  rccopse  cmpctoq  ot  error 

Meces&cy  ta  ptan  aterjua  > 

f  r-e  to  recow  from  error 

Tme  to  »edo  part  d  piar 

d  o >n*nr  astern  atatto 

Contproftenkon  c*  bank  subtree 

'Justly  of  deciBonmakro 

Table  5.4.  Examples  of  Time-  and  Accuracy-Based  Measures 
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1  MoPft  Tochnttal  Services  Attributes 

Hardware  and  Software 

f  A»aiat<it> 

1  ureteral  cop  abides  arable  to  jscra 

Ab4«y  to  v.ron?  partal  desloictcr  d  sistem 

RctousO'G^ErxIii'Bnc* 

to  adapt  i o  erwiroomerc 

mmalflaWiv 

Ease  of  ropalr  cx  reo  accmar  Ouiifl  operation 

CampuUrtcr  Capacity 

PcrteOMy 

i  Atotoy  to  operate  on  Offerer*  plBtkrms  I 

MetfcUy 

t  TfTTT'iv::  rr;.;T-' '  i 

1  MoPs  Technlc-ai  Sorviccs  -  Applications  Attributes 

intemperabmy 

Commumcaticrs  wtb  otrer  C2  systems 

Security 

Confrtecbatly  »*d  ifilftQrthr  of  data 

ConfidMitiAfcty 

litfo'melkjr'i  protected  el  eptnopiiiite  level 

inter*, 

Requred  kv  ixy/itlanr*  of  date 

CuMormiitxtty 

• tatty  to  custom  se  paremete-s  »  actus*  actrvites 

Ouorbtr  c ( Infonrebon 

Provide  at  ndormaten  reared  by  user 

BancMndlh 

1  MoCEs  Usor  Effectiveness  Information  Oualit# 

Setdcowty 

tatty  to  prevde  required  rriermater  n  requrad  mut 

The  ertent  to  frue  values  are  approached 

|  Conip'ihonMcn 

FucJitate  urdeisieiKjrQ  d  e*>ehon 

Response  tme 

T  inn  irw*** 

Information  aveisbte  M  appreyrote  tme 

pa*A  !:l  list* 

Training  time 

Decision  response  tme 

[  Time  available  to  commanders  1 

Table  5.5.  Examples  of  MoPs  and  MoCEs 


Example  Headquarters  C2  MoM 

C2  measures  may  also  be  divided  into  sets 
corresponding  to  the  sequential  steps  of  the  C2  cycle. 
These  include: 

•  Monitoring  and  understanding:  information 
transmission,  values,  times,  effect,  comprehension; 

•  Planning:  information  exchange,  coordination, 
impact,  flexibility,  process  quality;  and 

•  Directing  and  disseminating. 

The  MoM  for  C2  can  also  be  focused  on  four  levels:  a 
network  of  headquarters,  a  single  headquarters,  the 
individual  cells  within  the  headquarters,  and 
performance  of  specific  tasks  within  the  cells. 
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OOTWMoM 

While  national  NATO  policies  require  that  military 
forces  be  prepared  for  high  intensity  conflict,  forces 
have  been  increasingly  involved  in  low-intensity 
conflicts  and  C2  analyses  for  OOTW  are  therefore 
becoming  important.  OOTW  include  force  deployment 
to  create  or  maintain  conditions  for  a  political  solution 
in  order  to  avoid  escalation  into  hostilities.  Threats  to 
international  and  national  security  may  also  unfold  from 
natural  disasters,  terrorism  organised  crime,  civil 
unrest,  migration,  or  other  territorial  intrusions.  Most 
OOTW  are  inherently  joint  or  combined  operations. 

While  the  determination  of  MoM  has  been  stated  as 
difficult  to  obtain,  OOTW  offer  even  greater  challenge 
for  MoM.  Traditional  MoFE  such  as  loss  exchange 
ratios,  combat  effectiveness,  or  duration  of  the 
campaign  are  rarely  applicable  to  OOTW.  In  such 
operations,  military  forces  may  play  important  roles 
but  political  concerns  may  limit  the  scope  of  imposable 
solutions.  Public  and  political  pressures  may  result  in 
shifts  in  the  selection  of  criteria  for  MoM  e.g.,  more 
emphasis  may  be  placed  on  personnel  casualties  and 
less  on  equipment  losses. 

OOTWMoPE 

While  MoFE  and  MoCE  provide  measures  of  success 
for  military  operations,  MoPE  measure  the  degree  of 
attaining  political  objectives.  In  some  cases,  such  as 
humanitarian  assistance  or  nation  building  MoPE  may 
measure  the  degree  of  improvement  in  the  quality  of 
life  of  the  populace. 
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MoFE  usually  were  the  highest  MoM  used  within  the 
analysis  of  Article  V  missions3  assuming  that 
effectiveness  is  directly  related  to  the  higher  level 
MOPE  such  as  “winning  the  war.”  However,  such  an 
assumption  may  not  always  apply  to  OOTW.  For 
example,  military  actions  that  would  be  highly  effective 
in  accomplishing  mission  objectives  in  war  might  be 
quite  counterproductive  in  OOTW.  In  fact,  the  value 
of  military  actions  in  OOTW  is  not  so  much  a  question 
of  physical  effects,  but  rather  how  military  actions  and 
their  physical  effects  are  perceived  by  the  various 
actors  and  the  population  in  the  theatre,  how  the 
military  actors  interpret  the  behaviour  of  the  other 
actors  and  the  critical  mission  task  conditions  such 
as,  for  example,  political  interest  and  media  attention. 

McCafferty  and  Lea  developed  low-level  military-related 
measures  (MoCE)  to  cover  OOTW  (McCafferty  and  Lea, 
1 997).  The  MoM,  which  are  classified  as  MoCE,  include  the: 

•  Time  between  the  arrival  of  friendly  forces  in  the 
area  and  their  deployment; 

•  Time  between  deployment  of  friendly  forces  and 
contact  with  adversary  forces; 

•  Length  of  time  adversary  forces  were  under 
observation  without  posing  a  threat  to  friendly  forces; 

•  Length  of  time  friendly  forces  are  in  potential 
danger  (i.e.,  adversary  forces  have  the 
opportunity  to  fire  on  friendly  forces);  and 

•  Time  horizon  of  friendly  C2  processes  (how  far 
into  the  future  they  are  focused). 
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Mobility  may  be  important  for  OOTW,  as  well  as 
sustainability  and  self-sufficiency  in  theatre,  with  the 
implication  of  emphasising  measures  of  reliability  and 
maintainability.  Moreover,  the  perception  of  the 
capabilities  of  deployed  forces  acts  as  deterrence  or 
coercion  on  the  parties  in  conflict. 

Some  examples  identified  by  McCafferty  and  Lea  are: 

•  Opportunities  to  employ  forces,  which  reflects  the 
range  of  military  capabilities  available; 

•  Strategic  deployment,  which  is  related  to 
deploying  and  recovering  the  right  force  to 
theatre  efficiently  and  in  time; 

•  Endurance,  to  maintain  an  effective  force  in 
theatre  for  an  extended  time; 

•  Mission  objectives,  to  measure  the  success  of 
achieving  military  objectives  in  OOTW;  and 

•  Successful  termination,  to  deal  with  progress  to 
the  desired  end  state  (the  criteria  may  be  political 
and  thus  not  measured  by  military  activities). 

One  class  of  effectiveness  indicators  in  OOTW  is 
provided  by  transition  measures,  which  focus  on  the 
progress  in  the  transfer  of  responsibilities  to  the  follow- 
on  military  force  or  civil  agency.  T ransition  measures 
focus  on  the  degree  that  follow-on  organisations 
assume  tasks  and  responsibilities. 

Progress  toward  success  may  be  tracked  by  normality 
indicators,  which  are  indirect  measures  of  the  effects 
of  military  involvement  in  OOTW,  although  causal 
relationships  are  difficult  to  prove  (Lambert,  2000). 
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These  MoPE  may  be  obtained  by  evaluating  the  extent 
to  which  conditions  have  been  restored. 

Normality  indicators  measure  the  level  of  improvement 
in  the  quality  of  life  of  the  general  population,  and  may 
be  defined  as  “relative  measures  of  the  state  of 
normalcy  characterising  an  element  of  the  civil 
environment,  through  data  collected  on  a  regular  basis 
and  assessed  to  have  the  frequency,  quantity, 
consistency  and  coverage  required  to  make  a  useful 
objective  assessment  of  the  changes  occurring  in  the 
civilian  populace.”  (Department  of  National  Defence, 
Canada,  1999).  Normality  indicators  can  be  grouped 
in  categories  and  adapted  to  meet  the  changing 
requirements:  political,  socio-economic  levels  of 
development,  cultural,  legal  and  technological. 


Criterion 

Example* 

Political 

Elections,  political  participation 

Economic 

Unemployment  interest  rates,  and  market  baskets 

Socsal 

Number  of  stuctarts  in  schools  number  of  refugees 

lectinoloaical 

1  elenlrofie  system  availetxlity 

Legal 

Judicial  system  functioning 

Envronmerrtai 

Roads  water  supply,  power  supply 

Cultural 

Sports  events  concerts 

Table  5.6.  Normality  Indicators 


Limitations  of  normality  indicators  include: 

•  Inexperienced  personnel  available  for  data 
collection  and  analysis.  A  mix  of  inexperienced 
civil  and  military  personnel  are  often  assigned  to 
collect  data  in  fields  foreign  to  them;  training  may 
be  necessary  to  assure  reliable  and  valid  data; 

•  Temporary  effect  due  to  military  presence  for  data 
collection.  The  mere  presence  of  military  personnel 
collecting  data  may  affect  normality  measurements; 

•  Difficulties  in  obtaining  valid  and  reliable  data 
calibrated  against  baselines.  It  may  be  difficult  to 
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establish  the  threshold  for  “normality”  if  archive 
data  is  not  available; 

•  Extrapolation  in  space  and  time  from  a  specific 
locality.  It  may  be  inappropriate  to  extrapolate  civil 
progress  to  an  entire  region.  Sampling  is  important; 

•  Limited  resources  and  constraints  for  collection 
and  analysis.  Data  must  be  collected 
consistently  but  may  be  occasionally 
unobtainable  due  to  physical  inaccessibility  or 
lack  of  personnel; 

•  A  “snapshot”  which  may  not  provide  trends  if 
infrequently  obtained.  Trend  analysis  requires 
sufficient  data  and  sampling  rates;  and 

•  During  OOTWs,  the  relevant  MoM  may  change 
over  time,  particularly  the  lower  level  MoCE  and 
MoP.  For  example,  during  the  earliest  phases  of 
NATO  operations  in  Bosnia,  tracking  weapon 
systems  and  knowing  how  many  of  them  were  in 
cantonments  was  a  major  MoM.  Once  the  forces 
were  separated  and  most  weapon  systems  under 
control,  emphasis  shifted  to  the  disruptive  activities 
such  as  road  blocks  and  ethnic  harassment.  As 
these  behaviours  became  less  frequent,  NATO’s 
emphasis  shifted  to  normality  indicators. 

MoM  Hierarchies:  Some  Examples 

Modular  Command  and  Control  Evaluation  System 

Evaluation  of  C2  effectiveness  requires  a 
comprehensive  approach  for  the  preparation  of  the 
evaluation  process,  the  collection  of  data,  and  its 
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interpretation.  MCES  addresses  both  the  managerial 
and  analytical  aspects  of  evaluation  and  was  originally 
developed  for  the  systematic  comparison  of  C2 
systems.  The  objective  of  MCES  is  to  guide  analysts 
in  the  identification  of  appropriate  measures  for 
estimating  the  effects  of  C2  on  combat. 

MCES  prescribes  a  process  of  measurement,  but  does 
not  identify  either  a  measurement  system  or  a  set  of 
measures.  Similarly,  while  calling  for  the  collection  of 
data,  MCES  does  not  provide  details  on  how  data  are 
to  be  collected.  MCES  does  provide  guidance  on  how 
good  measures  and  good  collection  procedures  are 
characterised,  but  leave  the  details  of  the 
measurement,  data  collection,  and  analysis  plans  to 
the  analyst. 

MCES  considers  C2  as  consisting  of  three  components: 

•  Physical  entities  (equipment,  software,  people); 

•  Structure  (interrelationships  of  entities);  and 

•  Processes  (C2  functions). 

The  boundary  of  a  C2  system  here  is  defined  as  a  delineation 
between  the  system  studied  and  the  environment.  The  US 
Army’s  Training  and  Doctrine  Command  (TRADOC)  C2 
MoCE  Handbook  adds  mission  objective  as  the  top  layer 
of  the  hierarchy  of  C2  components. 

MCES  focuses  on  measures  as  opposed  to  models, 
but  includes  the  cybernetic  loop  model  of  generic  C2. 
It  consists  of  seven  procedural  steps. 
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HEAT 

While  originally  developed  for  theatre-level  combat 
applications,  the  HEAT  system  has  proven  robust.  For 
example,  it  has  been  used  to  assess  US  Department 
of  State  crisis  task  force  performance,  military 
operations  in  Grenada,  Panama,  and  Haiti,  and  several 
exercises  focused  on  peacekeeping,  and  humanitarian 
assistance  missions.  The  underlying  C2  process  of 
monitoring,  understanding,  and  developing  alternative 
actions;  predicting  the  consequences  of  each 
alternative  for  each  possible  future  under 
consideration,  decision;  developing  and  promulgation 
of  plans  and  directives  and  requests  for  support; 
seeking  information;  making  reports;  and  responding 
to  inquiries  are  all  relevant  across  the  ranges  of  OOTW 
missions.  HEAT  has  been  modified  since  2000  to 
include  measure  of  the  quality  of  collaboration,  which 
is  often  a  key  process  in  OOTW. 

ACCES 

ACCES  is  a  derivation  of  HEAT,  which  was  developed 
primarily  for  joint  theatre-level  operations.  ACCES 
reorganised  HEAT  concepts  into  army  doctrinal 
language  and  doctrine,  but  shares  the  same 
philosophy.  ACCES  has  been  applied  to  numerous 
division  and  corps  command  centre  assessments.  It 
represents  a  comprehensive  set  of  practical  and 
objective  performance  measures  for  C2  activities.  The 
primary  focus  of  ACCES  is  the  overall  performance  of 
a  command  centre  or  network  of  command  centres, 
at  various  stages  of  the  C2  process,  from  the  collection 
of  data  to  the  conversion  of  data  to  intelligence  to  the 
implementation  of  plans  and  directives.  The  underlying 
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approach  to  ACCES  is  that  C2  comprises 
interdependent  sub-processes  which  can  be  observed 
and  measured.  ACCES  considers  C2  as  an  adaptive 
control  process,  where  information  collected  from  the 
outside  is  processed  internally  to  generate  plans  that 
may  be  adapted  to  reflect  new  information.  ACCES 
takes  the  view  that  the  overall  effectiveness  of  a 
command  centre  can  be  judged  by  the  viability  of  its 
plans.  A  good  plan  is  one  that  can  be  executed  without 
the  need  for  modification  beyond  the  contingencies 
stated  in  the  plan  and  that  remains  in  effect  throughout 
its  intended  life. 

Multi-Attribute  Utility  Theory  (MAUT) 

MAUT  is  similar  to  ACCES  in  the  sense  that  both  use 
functional  decomposition  and  function-specific 
evaluation  metrics.  The  major  differences  are  that  MAUT 
can  be  used  with  any  set  of  metrics  (including  those 
from  ACCES),  which  must  be  specified  by  the  analyst. 
MAUT  assigns  weights  to  the  MoM  at  each  level  of  the 
MoM  hierarchy  and  utility  values  or  scores  at  the  lowest 
level.  MAUT  then  aggregates  upwards  the  weighted 
scores  to  provide  composite  scores  of  effectiveness. 
MAUT,  if  properly  used  with  appropriate  application  of 
judgmental  weights,  will  allow  integrated  analyses  based 
on  multiple  MoM.  While  this  is  often  satisfying  to 
decisionmakers  (it  provides  a  single  index  of  quality), 
analysis  should  always  monitor  the  components  of  such 
indices  as  they  may  provide  insight  into  strength  and 
weaknesses  of  the  C2  system.  For  example,  many 
OOTW  C2  problems  involve  a  variety  of  objective 
functions  and  trade-offs.  MAUT  results  should  always 
be  assessed  by  sensitivity  analyses. 
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Many  applications  of  MAUT  assume  additive 
composition  of  MoM.  However,  this  is  a  very  restrictive 
assumption  that  needs  to  be  validated  in  each  case. 
In  addition  MoM  may  interact  suggesting  that  the 
aggregation  of  MoM  is  at  partly  multiplicative  (Keeny 
and  Raiffa,  1976;  Sarin,  2000). 

Collaboration  C2  Metrics 

The  following  collaboration  metrics  have  evolved  out 
of  work  done  by  Evidence  Based  Research,  Inc.  for 
the  United  States  Office  of  Naval  Research.  These 
collaboration  metrics  focus  on  individual  and  team 
cognitive/awareness,  team  behaviour,  and  team 
products.  Individual  cognitive  metrics  measure 
collaboration,  team  members’  understandings  about 
their  mission  and  their  team,  and  team  cognitive 
metrics  apply  the  individual  cognitive  metrics  to 
quantify  the  level  of  awareness  in  a  team.  There  are 
four  classes  of  these  metrics: 

•  Averages  of  the  understanding  among  team  members; 

•  Extent  of  alignment  of  these  understandings; 

•  Maximum  level  of  understanding  anywhere  within 
the  team;  and 

•  Presence  of  gaps  in  understanding  throughout 
the  entire  team. 

Team  behaviour  metrics  measure  the  key  behaviours 
indicative  of  effective  teams.  These  behaviours  include 
smooth  and  efficient  synchronisation,  efficient 
information  exchange,  adaptability,  effective  workload 
distribution,  and  team  member  engagement.  Team 
product  metrics  measure  product  quality  and  team 
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efficiency  and  are  the  bottom-line  “proof  of  the  pudding” 
metrics,  applicable  whether  a  team  or  a  single 
individual  produces  the  product. 

Other  metrics  under  consideration  are:  task 
performance,  workload,  level  of  engagement  (buy-in), 
synchronisation,  information  needs  workload  and 
handling,  workload  awareness  and  handling,  and 
problem  awareness  and  handling. 

Other  Considerations  in  the 
Selection  and  Interpretation  of  MoM 

Effects  of  Uncertainty 

In  order  to  state  a  level  of  confidence  in  the 
interpretation  of  MoM,  the  underlying  assumptions 
must  be  clearly  stated  and  uncertainties  recognised. 
Uncertainties  manifest  themselves  in  several  ways  that 
may  affect  MoM.  They  may  be  grouped  as  follows: 

•  Study  assumptions — (uncertainties  in  the 
scenario,  model  input); 

■  Relevance  to  the  purpose  of  the  evaluation, 
uncertainties  in  the  military  objective, 
knowledge  of  enemy  concept  of  operations, 
intentions,  capabilities,  weapon  performance, 
uncertainties  in  terrain  data,  etc. 

•  Modelling  assumptions — (uncertainties  in  the 
model,  structural  uncertainty);  and 

■  Human  performance,  parameters,  objects, 
attributes,  processes,  effects  of  constraints, 
effects  of  aggregation  and  de-aggregation, 
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deterministic  (usually  high  hierarchical  level 
but  low  resolution)  versus  stochastic  models, 
especially  in  OOTWs;  and 

■  Uncertainties  about  the  implications  of  value 
changes  in  lower-level  MoM  with  respect  to 
the  values  of  higher-level  MoM  (e.g., 
increases  in  arrest  rates  at  late  stages  of  an 
OOTW  may  be  negative,  while  increases  in 
arrest  rates  at  early  stages  may  be  positive). 

•  Model  sensitivity — (uncertainties  in  the  outcome) 

■  Hypersensitivity  to  input  variations,  (instability 
or  chaos  theory),  effects  of  model  non- 
linearities  and  non-monotonic  behaviour 
(effects  of  thresholds),  decisionmaking  for 
local  versus  global  optimisations,  etc.;  and 

■  Sensitivity  analysis  may  be  applied  to  identify 
uncertainty.  By  varying  the  assumptions  and 
input  data  within  the  plausible  ranges, 
excursions  in  the  analysis  verified  by  the 
subject  matter  experts  provide  insight  into  the 
effects  of  uncertainty. 

Impact  of  Technological  Changes 

The  rapid  pace  of  technological  change  involving 
information  systems  is  causing  major  changes  in  the  way 
C2  is  perceived  and  executed,  leading  to  potential  changes 
in  the  way  war  fighting  commands  are  organised.  For 
example,  the  last  decade  has  seen  the  emergence  of 
collaborative  technologies,  which  enable  new  ways  to 
command  and  control  military  forces  and  for  them  to 
interface  with  other  actors.  To  keep  pace  with  and  to 
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evaluate  the  impacts  of  these  changes,  the  nature  of  these 
changes  and  impacts  need  to  be  understood  so  that  the 
appropriate  MoM  can  be  developed. 

One  approach  to  doing  this  postulates  those 
differences  likely  to  occur  between  today’s  C2  and 
future  C2,  and  then  describes  an  evaluation 
methodology,  including  MoM,  to  measure  the  impact 
of  the  changes.  Other  approaches  are  Social 
Construction  of  Technology  (SCOT)  (Bijker,  1989), 
Socio-Technical  Networks  (Eizen,  Enserink  and  Smit, 
1 996)  and  Actor  Network  Theory  (ANT)  (Latour,  1 997). 

Conclusions 

No  single  measure  or  methodology  exists  that 
satisfactorily  assesses  the  overall  effectiveness  of  C2 
systems.  As  a  minimum,  the  following  factors  must  be 
considered  in  conducting  an  analysis  of  C2: 

•  Determine  the  appropriate  levels  of  MoM  hierarchy; 

•  Identify  specific  MoM  which  are  practically  obtainable; 

•  Specify  means  of  collection  of  MoM; 

•  Assure  the  validity  and  reliability  of  measures  for  correct 
interpretation  with  quantifiable  levels  of  confidence; 

•  Be  aware  that  variation  in  measurements  (e.g., 
due  to  human  factors)  may  well  cause 
unacceptable  levels  of  uncertainty.  Hence  the 
analyst  must  pay  particular  attention  to 
measurements  related  to  the  human  element; 


•  Consider  that  while  MoPE  and  MoFE  may  provide 
the  most  persuasive  measures  from  the  military 
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perspective,  MoCE  and  MoP  are  the  most  readily 
derivable  by  operations  analysts;  and 

•  Account  for  the  principles  of  reliability  and  validity 
to  avoid  the  risk  of  generating  false  conclusions. 

Recommendations  for  Generation  and 
Selection  of  MoM 

The  principal  objective  for  MoM  is  to  determine 
judgements  of  the  degree  to  which  C2  or  changes  to 
C2  may  improve  force  effectiveness  and  to  provide 
convincing  arguments  for  the  improvements.  It  is 
important  to  stress  that  the  purpose  is  to  assess  the 
contributions  of  C2  in  terms  of  how  C2  improves  the 
effectiveness  of  military  missions,  and  not  the  quality 
of  the  C2  process  itself.  However,  to  arrive  at  these 
assessments  and  assign  attribution  to  the  C2  system, 
the  C2  system  must  be  included  in  the  analysis.  To 
achieve  this  objective,  the  following  steps  are  required: 

•  The  objectives  for  the  assessment  must  be 
established  and  clearly  stated; 

•  Selection  of  MoM  should  not  be  done  in  isolation 
from  consideration  of  the  assumptions, 
constraints,  models,  tools,  scenarios,  or  other 
elements  of  the  analytic  plan  and  assessment 
process.  The  assumptions  used  in  the  model 
and/or  evaluation  must  be  stated  along  with  their 
potential  impact  on  the  results; 

•  A  detailed  assessment  of  reliability  and  validity  of 
the  selected  measures  needs  to  be  made  in 
order  to  determine  a  level  of  confidence  in 
measures;  and 
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•  For  C2  acquisition  analyses,  the  generation  of 
measures  should  occur  in  parallel  with  the 
development  of  the  system,  so  that  as  the 
system  is  being  matured,  developers  can  know 
the  standards  to  which  they  are  being  held. 

Summary  of  the  Challenges  and  Issues 
in  the  Evaluation  of  C2 

•  Correlation  of  MoPE  and  MoFE  with  C2  process 
measures  (e.g.,  battle  outcome  against  lower- 
level  measures)  is  difficult; 

•  Separation  and  linkage  of  the  respective 
relationships  between  C2  and  users, 
organisations  and  military  objectives  requires 
some  effort; 

•  Aggregated  measures  (e.g.,  as  obtained  from 
ACCES  or  MAUT)  have  limitations  in  the 
diagnosis  of  C2  success  or  failure.  A  careful 
analysis  is  required  to  provide  a  comprehensive 
assessment  of  highly  complex  C2  systems 
based  on  a  small  number  of  summary  measures 
of  outcome  and  process; 

•  The  assessment  of  the  reliability  of  measures  in 
an  environment  where  sample  sizes  are  small 
will  remain  difficult  and  may  require  the  use  of 
non-parametric  statistics; 

•  The  analyst  must  pay  attention  to  the  complex 
task  of  establishing  and  measuring  control 
variables  in  order  to  achieve  correlation  of 
measures  against  a  wide  spectrum  of  scenarios 
and  staff; 
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•  Defining  criteria  to  differentiate  measures  must 
be  established; 

•  Verifying  measurement  criteria  (e.g., 
discrimination)  must  be  ensured; 

•  For  the  near  future,  collecting  data  to  support  C2 
measures  will  remain  labour  intensive  because 
C2  processes  remain  human  intensive; 

•  Many  of  the  measures  for  information  processing 
concern  completeness  of  the  information. 
Deciding  what  makes  information  complete 
requires  coordination  and  cooperation  between 
the  assessor  and  the  user; 

•  The  relationship  between  outcome  and  process 
may  be  complex  because  C2  is  an  integrated 
system  with  continuous  feedback; 

•  The  analysis  of  uncertainties  and  measures  of 
central  tendency  and  dispersion  are  both 
significant  when  examining  C2  issues;  and 

•  A  Command  Post  Exercise  (CPX)  is  a  useful 
venue  for  evaluation  of  C2.  However,  the  costs 
involved  generally  preclude  conducting  a  CPX 
solely  to  evaluate  the  C2  process  or  a  C2  system. 
Cost  control  is  increasingly  leading  to  the  use  of 
laboratory  and  human  in  the  loop  experimentation 
to  develop  knowledge  and  insights. 

Summary  of  the  Challenges  and  Issues 
forOOTWMoM 

•  Cost/benefit:  cost  in  time  and  effort  for  data 
collection  and  analysis  may  outweigh  benefit; 
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•  Standardisation  of  data  collection,  evaluation, 
and  analysis  with  many  diverse  non¬ 
governmental  organizations  (NGOs),  UN 
agencies,  and  militaries; 

•  Factors  outside  military  control:  other  agencies, 
policy  restrictions; 

•  Creation  of  a  rich  and  comprehensive  set  of  MoM 
to  preclude  reliance  on  a  limited  number  of  MoM 
as  the  key  to  success; 

•  Availability  and  consistency  of  information  (e.g., 
in  OOTW,  a  possible  consequence  of  different 
factions  controlling  geographic  areas); 

•  Merging  of  strategic,  operational,  and  tactical 
domains;  and 

•  Clear  recognition  of  roles  and  responsibilities  of 
all  participants. 

Chapter  5  Acronyms 

ACCES-  Army  Command  and  Control  Evaluation  System 

AHWG  -  AC/243  Panel  7  Ad  Hoc  Working  Group 

ANT  -  Actor  Network  Theory 

C2  -  Command  and  Control 

C3I- Command,  Control,  Communications  &  Intelligence 

COBP  -  Code  of  Best  Practise 

CPX  -  Command  Post  Exercise 

DP  -  Dimensional  Parameters 
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HEAT  -  Headquarters  Effectiveness  Assessment  System 

MAUT  -  Multi-Attribute  Utility  Theory 

MCES  -  Modular  Command  and  Control  Evaluation  Structure 

MoCE  -  Measures  of  C2  Effectiveness 

MoE  -  Measures  of  Effectiveness 

MoFE  -  Measures  of  Force  Effectiveness 

MoM  -  Measures  of  Merit 

MoP  -  Measures  of  Performance 

MoPE  -  Measures  of  Policy  Effectiveness 

MORS  -  Military  Operations  Research  Society 

NGO  -  Non-Governmental  Organization 

OOTW  -  Operations  Other  Than  War 

SCOT  -Social  Construction  of  Technology 

TRAC  -  TRADOC  Analysis  Center 

TRADOC  -  US  Army  Training  &  Doctrine  Command 
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should  concentrate  almost  entirely  on 
the  ‘actualities  of  war,  ’  the  effect  of 
tiredness,  hunger,  fear,  lack  of  sleep, 
weather. .  .It  is  the  actualities  that  make 
war  so  complicated  and  so  difficult,  and 
are  usually  neglected  by  historians. 

— Field  Marshall  Archibald  Wavell,  1 883-1 950, 
Author  of  Soldiers  and  Soldiering 
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Importance  of  Human  and 
Organisational  Factors 

The  human  dimension  largely  distinguishes 
command  and  control  (C2).  Key  differences 
between  C2  analyses  and  traditional  military 
operations  analysis  (OA)  applications  include  the  need 
to  deal  not  only  with  military  organisations,  but  also 
with  distributed  military  teams  (and  organisations) 
under  stress  and  their  decisionmaking  behaviour  as 
well.  Moreover,  in  operations  other  than  war  (OOTW), 
consideration  must  be  paid  to  the  behaviour  of  and 
interaction  with  non-military  organisations,  political 
groupings,  and  amorphous  groups  such  as  crowds  and 
refugees.  Thus,  the  formulation  of  the  problem  and 
the  development  of  solution  strategies  cannot  be 
completed  without  explicit  consideration  of  both  human 
and  organisational  issues. 

The  human  factors  of  interest  fall  into  three  major  categories: 

•  Human  behaviour  related  to  performance 
degradation,  such  as  stress  and  fatigue,  and  as 
a  consequence  of  social  interactions  among 
individuals  and  members  of  groups; 

•  Decisionmaking  behaviour  (cognitive  questions) 
including  the  cognitive  complexity  of  the  issues 
and  the  capacities  of  the  commanders  or  other 
decisionmakers  of  interest;  and 

•  Command  style. 

By  contrast,  organisational  factors  deal  with 
relationships  among  groups  of  individuals,  including 
connectivity,  roles,  and  organisational  structures. 
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Since  both  human  and  organisational  factors  can  affect 
C2  performance,  the  operations  analyst  must  consider 
their  impact  early  in  the  research  design  process  and 
review  a  priori  assumptions  about  them  in  an  iterative 
manner  throughout  the  entire  analytical  process. 
Human  and  organisational  factors  must  be  considered 
as  part  of  structuring  the  problem,  selecting  measures 
of  merit  (MoM),  defining  scenarios,  developing  solution 
strategies,  and  selection  of  tools. 

The  first  key  consideration  when  structuring  the 
problem  is  whether  individual  decisionmaking  and 
behaviour  of  individuals  or  groups  is  important  to  the 
C2  processes  under  analysis.  If  the  research  question 
can  be  answered  without  considering  differences 
between  individual  decisionmakers  and  groups  then 
the  additional  complexity  that  issue  introduces  should 
be  avoided.  For  example,  in  addressing  a  C2  issue 
that  deals,  all  other  things  being  equal,  with  a  simple 
change  in  connectivity  (which  headquarters  will  have 
which  linkages  to  others),  human  behaviour  may  not 
be  important  to  the  analysis  at  least  as  long  as  combat 
missions  are  involved.  However,  the  same  change  in 
connectivity  might  affect  relations  with  non-military 
organisations  and  individuals  essential  for  mission 
success  in  OOTW  suggesting  their  response  must  be 
accounted  for  in  an  appropriate  manner.  Thus, 
deciding  on  the  importance  of  human  issues  in  OOTW 
and  identifying  the  issues  at  stake  requires  a  good 
general  understanding  of  human  behaviour  and  its 
underlying  motivations  beyond  military  common  sense 
as  well  as  knowledge  about  relevant  cultural  factors 
and  the  interests  of  the  parties  involved. 
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Human  Factors 

Human  Performance  and  Behaviour 

Human  performance  affects  behaviour  and  vice  versa. 
Human  performance  depends  on  psycho-physiological 
variables  (e.g.,  stress,  fatigue,  sleep  deprivation,  hunger, 
and  alertness)  and  on  ergonomic  and  external  factors 
limiting  performance  and  behavioural  freedom. 
Individual  and  group  behaviour  is  the  result  of  social 
interaction.  It  includes  interactions  by  military 
commanders  and  their  troops,  underlying  psychological 
processes  and  factors  (e.g.,  fear,  morale,  and  values), 
and  the  cultural,  educational,  and  religious  background 
of  individuals.  There  is  significant  historical  evidence 
that  inferior  combat  potential  as  measured  in  terms  of 
numbers  of  personnel  and  weapon  systems  may  be 
compensated  for  by  superior  human  performance  in 
battle  (Dupuy,  1979). 

Any  time  human  performance  and/or  behaviour  are  at 
issue,  parameters  and/or  models  will  be  needed  to 
reflect  those  issues.  For  example,  systems  that  involve 
human  activity,  such  as  watch  or  command  centres, 
need  to  be  studied  in  ways  that  reflect  differences  in 
C2  performance  that  can  be  traced  to  human 
performance/behaviour  issues.  In  addition,  differences 
can  arise  from  experience  or  training,  coalition  features 
(e.g.,  language,  national  doctrine,  command  style),  or 
service/branch-unique  doctrine  and  practice.  These 
kinds  of  individual  performance  and  behaviour  issues 
may  be  modelled  in  two  ways.  They  can  be  treated 
stochastically,  in  a  manner  that  reflects  their 
occurrence  in  “real”  systems  depending  on  situational 
factors  (black  box  approach),  or  in  terms  of  process- 
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oriented  behavioural  models,  describing  the 
psychological  processes  behind  the  observable 
behaviour  of  individuals  or  groups  in  a  given  situation 
(Long  Term  Scientific  Study  SAS-017  on  Human 
Behaviour  Representation,  Final  Report,  Chapter  2). 
The  decision  that  human  performance  and  behaviour 
may  vary  meaningfully  will  have  a  clear  impact  on  the 
choice  of  models  and  analytic  approaches  (e.g., 
stochastic  processes  or  action-theoretic  models  such 
as  Norman’s  Activation  Trigger  Scheme  (ATS)  for 
simulating  the  dynamics  of  actions  or  reactions) 
(Norman,  n.d.). 

Where  human  performance  is  considered  to  be  a 
meaningful  factor  (e.g.,  C2  within  command  centres 
when  wearing  chemical  protection  gear),  some 
experimentation  may  be  necessary  to  develop  realistic 
parameters  for  the  impact  on  error  rates  or  the  pace 
of  work.  In  other  cases,  such  as  simple  fatigue,  human 
factors  specialists  may  be  able  to  provide  valid 
parameters  from  work  in  other  contexts.  Such 
specialists  are  often  valuable  members  of  research 
teams.  In  any  case  where  the  workflow  and  work  rate 
within  command  centres  is  relevant,  human 
performance  parameters  must  be  considered.  In 
addition,  human  performance  issues  will  have  some 
effect  on  decisionmaking.  Error  rates  increase  as 
people  become  tired  and  overloaded,  altering  the  way 
they  work  and  the  information  they  consider. 

However,  research  on  how  to  represent  human 
behaviour  and  its  impact  on  performance  in  models  is 
still  in  an  early  stage.  This  is  particularly  true  in  the 
context  of  OOTW.  Those  operations  require  the  co¬ 
operation  of  non-military  actors.  In  addition,  military 
mission  objectives  include  providing  security  and 
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assistance  in  the  reconstruction  of  conditions  in  the 
theatre  of  operations  and  helping  to  stabilise  the 
situation  to  a  degree  that  ultimately  permits  the  military 
forces  to  leave.  The  use  of  force  in  OOTW  is  limited  in 
degree  and  kind  to  responding  appropriately  to  threats 
for  defensive  and  protective  purposes  and  to  coercive 
actions  for  enforcing  compliance  with  agreements. 
Military  actions  that  would  be  highly  effective  in 
accomplishing  mission  objectives  in  war  might  be 
counterproductive  in  OOTW.  In  fact,  the  value  of 
military  actions  in  OOTW  is  not  so  much  a  question  of 
physical  effects,  but  rather  how  military  actions  and 
their  physical  effects  are  perceived  by  the  various 
actors  and  the  population  in  the  theatre,  how  the 
military  actors  interpret  the  behaviour  of  the  other 
actors,  and  the  critical  mission  task  conditions  (e.g., 
political  interest  and  media  attention).  Thus,  a  careful 
analysis  of  mission  tasks  by  human  science  experts 
is  indispensable  for  modelling  and  assessment  of  C2 
options  in  OOTW  (Baeyer,  2001 ). 

It  follows  that,  in  addition  to  the  traditional  composition 
operations  research/operations  analysis  (OR/OA) 
assessment  teams,  assessors  of  C2  for  OOTW  must 
possess  hands-on  experience  with  such  operations 
and  relevant  non-military  organisations  (e.g.,  private 
aid  organisations).  The  assessment  team  must  also 
have  access  to  experts  from  the  fields  of  political 
science,  cultural  anthropology,  demography,  sociology 
(including  media  impact  research),  social  psychology, 
and  individual  psychology.  These  experts  will 
contribute  the  expertise  for  diagnosing  the  relevance 
of,  and  differences  in,  performance  and  behaviour  of 
actors  and  for  the  formulation  of  hypotheses  for 
assessing  the  “human  issue  risk”  of  analysis  results. 


Chapter  6  1 33 


In  addition,  they  may  contribute  to  the  modelling  of 
behavioural  processes  in  analysis  tools  used  to  test 
C2  system  sensitivity  via  parametric  variations  of 
human  performance  and  behaviour  parameters. 

However,  well-documented  empirical  knowledge  on 
human  performance  in  military  operations  is  scarce 
and  little  is  known  about  its  relevance  in  circumstances 
other  than  those  prevailing  when  the  underlying  data 
were  collected.  Similarly,  experience  and 
systematically  compiled  data  on  behaviour  and 
response  of  individuals  and  groups  to  actions  and 
situations  in  OOTW  are  still  limited  and  theories  on 
human  behaviour  are  mostly  untested  in  the  context 
of  military  operations.  Therefore,  dealing  with  human 
issues  in  C2  analyses  reduces  to  the  problem  of 
addressing  decisions  under  risk  and  uncertainty  when 
each  of  the  C2  design  options  is  tested  for  the  range 
of  possible  hypotheses  on  the  implications  of  human 
issues  for  C2  effectiveness. 

Human  Decisionmaking 

Increasingly,  assessment  teams  have  to  deal  with 
issues  where  individual  decisions  are  important.  This 
is  especially  true  for  OOTW  in  which  even  tactical- 
level  decisions  by  a  lower  level  military  leader  may 
have  strategic  implications  because  of  media 
presence.  This  represents  a  major  challenge  because 
the  variety  of  human  behaviours  involved  makes 
modelling  decisionmaking  very  difficult.  Fortunately, 
there  are  some  approaches  that  can  be  used  to  cope 
with  these  difficulties.  The  correct  choice,  however, 
will  depend  on  the  research  issue(s). 
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In  some  cases  the  analyst  is  asked  to  assume  that 
decisionmaking  will  follow  established  doctrine  and 
tactics,  techniques,  and  procedures  (TTPs).  In  these 
cases,  the  challenge  is  to  craft  a  set  of  rules  or  look  up 
tables  that  reflect  the  existing  guidance  correctly.  Hence, 
a  model  that  replicated  the  “correct”  set  of  decisions 
would  be  useful  for  assessing  simple  C2  issues  such 
as  the  impact  of  new  communication  and  information 
technology  or  changes  in  connectivity  or  supporting 
relationships  within  the  force.  However,  maximising  the 
benefits  of  technical  and  organisational  changes  in  a 
C2  system  might  require  an  appropriate  adaptation  of 
doctrine  and  TTPs.  Therefore,  testing  the  impact  of 
established  rules  should  be  part  of  the  analysis. 

Testing  the  impact  of  rules  is  an  indispensable 
prerequisite  for  models  that  have  built-in  sets  of  rules 
that  are  not  driven  by  approved  TTPs,  but  rather  by 
opinions  of  subject  matter  experts  or  modellers  whose 
rationales  have  been  neither  validated  nor  accredited. 
Considerable  knowledge  exists  on  how  to  organise 
and  validate  such  expert  elicitation.  Here  again, 
specialised  team  members  may  be  helpful.  Simple 
adoption  of  models  developed  from  subject  matter 
experts  will  put  the  assessment  team  at  considerable 
risk  of  accepting  false  conclusions.  When  such  models 
must  be  adopted,  they  should  be  explored  in  detail  to 
uncover  their  driving  assumptions  and  subjected  to 
sensitivity  analyses  (Chapter  9).  Where  this  cannot 
be  done,  these  models  are  best  avoided  when  C2 
assessments  are  performed. 

From  human  factors  research  on  stimuli  that  influence 
human  decisionmaking  we  have  learned  that  human 
decisionmaking  capability  is  degraded  in  some 
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situations  and  enhanced  in  others.  These  stimuli  may 
originate  from,  inter  alia: 

•  Biological/physiological  processes  (e.g.,  physical 
overexertion  [fatigue],  use  of  bio-chemical 
substances,  and/or  sensory  deprivation); 

•  Cognitive  sources  (e.g.,  confusion  arising  in 
unfamiliar  or  unknown  situations); 

•  Psychological  processes  (e.g.,  processes 
causing  emotions  and  stress); 

•  Social  processes  (e.g.,  group  dynamics  that 
coerce  or  reinforce  individual  decisionmaking 
depending  on  accepted  social  norms, 
organisational  infrastructure  and  procedures); 

•  Environmental  factors  (e.g.,  darkness,  austere 
and/or  uncomfortable  environments);  and 

•  The  decision  support  tools  and  technologies  that 
humans  use  (e.g.,  information  displays  and  decision 
support  software).  It  is  important  to  look  beyond 
technology  at  whether  or  not  human  decisionmaking 
is  improved,  or  even  constrained  in  some  cases,  by 
using  computerised  decision  aids  depending  upon 
their  functionality  and  configuration. 

Types  of  Decisions 

The  nature  of  the  decisions  being  supported  by  C2  systems 
will  also  enable  the  assessment  team  to  make  intelligent 
decisions  about  how  they  influence  the  analysis.  Three 
useful  decision  types  can  be  distinguished: 
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•  Automatable  decisions; 

•  Contingent  decisions;  and 

•  Complex  decisions. 

Automatable  Decisions 

Automatable  decisions  fall  into  the  category  of  “simple 
decisions.”  The  range  of  decision  options  is  finite  and 
known,  and  the  criteria  for  selecting  among  them  are 
clear.  Basic  sensor-to-shooter  decisions  are  simple 
decisions  that  are  usually  automated  for  the  sake  of 
timely  response  (e.g.,  anti-aircraft  or  missile  defence 
systems).  Similarly,  the  selection  of  patrol  routes, 
inspection  strategies,  and  many  logistics  decisions 
relevant  for  OOTW  can  be  automated.  For  example, 
scheduling  can  be  seen  as  an  optimisation  problem  in 
which  time,  space,  and  priorities  are  traded  off  to 
generate  a  “best”  answer.  Even  though  the  decision 
environment  is  constantly  changing  due  to  factors  such 
as  weather  and  mechanical  problems,  scheduling 
decisions  are  characterised  by  rules  and  algorithms. 
Models  of  automatable  decisions  of  this  kind  can  be 
built  relatively  easily. 

However,  where  the  C2  system  employs  humans  to 
make  these  choices,  some  error  rate  parameters  will 
be  needed  if  the  results  are  to  be  meaningful.  For 
example,  error  rates  may  increase  if  the  time  available 
to  make  a  decision  is  insufficient  or  physical  demands 
induce  fatigue.  Even  where  the  operational  concept 
calls  for  the  use  of  automated  systems,  the  analyst 
should  explore  the  quality  of  the  data,  information,  or 
knowledge  used  to  drive  the  process  and  the  likelihood 
that  humans  will  be  involved  in  collection  or  fusion. 
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In  these  fully  automatable  decisions  the  assumption 
is  that  “to  know  is  to  decide.”  In  these  cases,  if 
uncertainty  were  adequately  reduced,  the  correct 
course  of  action  or  decision  would  be  obvious.  In  that 
case,  decision  theory  classifies  the  problem  as  a 
“decision  under  certainty.”  These  problems  are  trivial 
except  for  the  determination  of  the  utility  function  in 
case  there  are  more  than  two  selection  criteria  that 
need  to  be  considered  and/or  when  constraints  need 
to  be  accounted  for  to  complete  the  analysis  (Keeny 
and  Raiffa,  1976). 

Contingent  Decisions 

The  next  level  of  decisionmaking  complexity  is  best 
thought  of  as  contingent  decisions.  These  are  cases 
where  the  commander  has  thought  through  the  situation 
and  developed  a  set  of  alternative  actions  or  decisions 
that  are  appropriate  to  the  situation,  but  further  information 
on  the  operational  environment  will  be  needed  to 
determine  which  is  the  proper  course  of  action.  In  other 
words,  “to  know  is  to  decide,  but  knowing  is  not  yet 
possible.”  In  some  NATO  countries  the  research 
community  terms  this  “opportunistic  decisionmaking.” 

In  most  cases  a  lack  of  clear,  precise  knowledge  is 
unavoidable.  For  example,  the  commander  in  a 
defensive  posture  may  recognise  that  the  adversary 
has  several  potentially  viable  options.  The  adversary 
may  not  even  know  which  alternative  he  will  choose.  In 
such  a  case,  the  defending  commander  would  both 
develop  courses  of  action  to  meet  likely  contingencies 
and  also  undertake  a  variety  of  information  collection 
activities  designed  to  provide  as  much  warning  as 
possible  when  the  attacker  selects  a  main  attack  option. 
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Modelling  contingent  decisions  is  much  more  difficult 
than  modelling  automatable  decisions,  but  is  similar 
in  that  an  underlying  set  of  rules  or  algorithms  still 
drives  the  process.  The  added  complexity  comes  from 
the  need  to  find  the  time  when  information  is  adequate 
to  select  one  of  several  actions.  The  best  models  for 
that  purpose  are  essentially  hypothesis  testing  models. 
They  align  information  about  the  operational 
environment  against  a  finite  set  of  alternative  futures 
and  perform  probability  calculations  to  determine  when 
the  commander  has  enough  confidence  to  act  or  to 
estimate  the  information  gain  in  terms  of  the  expected 
value  added  to  decisions  as  new  information  arrives 
(Sherrill,  1 996).  Information  entropy  is  also  a  valuable 
measure  of  the  information  state  of  the  commander.  It 
can  also  be  extended  to  a  measure  of  information 
dominance  (Perry  and  Moffat,  1997). 

Complex  Decisions 

Finally,  “complex”  decisions  are  very  difficult  to  model. 
These  require  the  decisionmaking  system  to: 

•  Recognise  when  a  decision  needs  to  be  made; 

•  Identify  the  relevant  set  of  options; 

•  Specify  the  criteria  by  which  they  will  be  judged;  and 

•  Determine  when  the  decision  will  be  made. 

Examples  of  complex  decisions  include  the  definition 
of  missions  at  the  operational  level,  decisions  to 
change  the  fundamental  activity  of  the  organisation 
(e.g.,  shift  from  the  offence  to  the  defence),  and  the 
process  that  creates  courses  of  action  in  response  to 
events  on  the  battlefield  or  in  OOTW.  Except  when 
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doctrinal  answers  are  available,  complex  decisions  are 
very  difficult  to  model  and  even  more  difficult  to  validate 
or  accredit.  Most  successful  efforts  dealing  with 
complex  decisions  have  used  “human-in-the-loop” 
techniques  and  relied  on  the  quality  and  variety  of 
experts  employed  for  reliability  and  validity.  Some 
promising  research  on  modelling  complex  decisions 
in  military  operations  has  been  completed  in  the  UK 
and  this  is  in  the  process  of  being  incorporated  as  the 
core  of  the  next  generation  of  closed  form  simulation 
models  of  conflict  being  developed  by  Defence  Science 
and  Technology  Laboratories  (DSTL)  (Moffat,  2000; 
Moffat,  2002)  Similar  research  on  tactical  decision 
automata  is  going  on  in  Germany  to  improve  the 
capability  of  simulation  systems  for  analysis  as  well 
as  training  and  staff  exercises  support  (Hofmann  and 
Hofman,  2000,  von  Baeyer,  2001). 

Command  Style 

Assessment  teams  often  encounter  the  argument  that 
decisionmaking  depends  on  the  “commander’s  style.” 
Moreover,  they  are  told,  systems  must  be  designed  to 
support  commanders  with  differing  styles.  Because  it 
is  an  elusive  and  multi-dimensional  concept,  command 
style  represents  a  challenge  to  modelling.  However, 
this  factor  can  be  accommodated  if  the  analyst  is  able 
to  develop  a  clear  concept  of  the  alternative  command 
styles  that  must  be  recognised  and  their  consequences 
for  military  decisionmaking. 

Attributes  of  the  Commander 

Differences  in  command  style  may  be  reflected  by 
appropriate  attributes  such  as  the  background  and 
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training  of  commanders,  their  decision  and  order  style, 
risk  tolerance,  and  operational  experience.  For 
example,  in  conjunction  with  field  experience,  the 
background  and  training  of  commanders  affect  the 
richness  of  their  understanding  of  the  military  situation 
and  their  capacity  to  influence  it. 

Organisational  Style 

Another  not  totally  unrelated  topology  deals  with  the 
degree  to  which  the  commander  uses  a  formal 
decomposition  of  the  situation  versus  a  holistic,  integrated 
vision.  The  decomposition  style  of  management  is 
associated  with  hierarchical  and  segmented  work,  as  in 
the  Napoleonic  or  classic  German  general  staff.  This 
heavily  structured  process  allows  centralised  control  and 
tight  coupling  between  the  structure  of  the  problem,  the 
structure  of  the  supporting  staff,  and  the  flow  of 
information  within  and  between  command  centres.  The 
classic  centralised  commander  imposes  his  style  on  the 
C2  process  and  impacts  key  organisational  issues  as 
well  as  decision  style. 

The  alternative  command  style  is  an  open  and  holistic 
one  in  which  senior  staff  and  commanders  from  related 
command  levels  are  directly  involved  in  a  broad 
development  of  courses  of  action  and  implementation 
plans.  This  more  open  process  also  has  implications 
for  the  information  flow  within  and  between  command 
centres.  While  decisions  are  still  made  authoritatively 
at  the  centre  (by  the  commander  or  senior  staff),  they 
tend  to  generate  loose  guidance  (mission  type  orders) 
and  to  enable  lower  level  commanders  and  their  staffs 
more  latitude  in  implementation. 
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Risk  Style 

Another  topology  applied  by  some  practitioners  is  the 
degree  to  which  individual  commanders  (and 
doctrines)  are  risk  averse  versus  attracted  to  risk.  Most 
military  enterprises  have  some  properties  that  impel 
commanders  to  minimise  risk.  The  fact  that  lives, 
national  treasure,  and  serious  national  interests  are 
involved  in  warfare  suggests  that  risk  averse  strategies 
will  tend  to  dominate.  However,  some  military 
commanders  are  more  comfortable  with  greater  risk. 
Indeed,  outnumbered  or  otherwise  disadvantaged 
forces  must  often  take  risks  in  order  to  prevail.  To  the 
extent  that  the  relative  risk  aversion  of  commanders 
is  relevant  to  the  C2  analyses  underway,  assessment 
teams  will  need  to  define  and  model  variables  that 
represent  this  factor  (Schultz,  n.d.). 

Recent  research  on  Bayesian  decisionmaking  had 
indicated  that  the  perception  of  risk  as  measured  by  utility 
loss  relative  to  a  goal  value  and  perception  of  future 
outcomes  can  in  combination  give  insight  into  the  way 
these  affect  command  decisionmaking  (Moffat,  2002). 

Orders  Style 

Commanders,  and  national  command  styles,  have  also 
been  shown  to  differ  in  the  degree  of  detail  contained 
in  directives  to  subordinates.  At  one  end  of  the 
spectrum  is  the  commander  who  issues  detailed  orders 
that  specify  what  is  to  be  done,  how  it  is  to  be 
accomplished,  and  when  and  where  the  specified 
activities  are  to  occur.  At  the  opposite  end  of  the 
spectrum  is  the  commander  who  issues  “mission  type 
orders”  which  simply  specify  the  mission  to  be 
accomplished  and  leave  decisions  about  the  detailed 
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objectives,  forces  to  be  employed,  critical  terrain,  and 
timing  up  to  the  subordinate  commanders.  In  between 
are  those  who  specify  a  series  of  linked  objectives 
(cross  the  river,  take  the  high  ground  in  the  north,  and 
be  prepared  to  defend  or  carry  the  attack  north-east 
into  the  valley)  and  supporting  detail  (e.g.,  forces 
available,  rough  timetable  keyed  to  the  objectives)  but 
leave  subordinates  with  considerable  discretion  within 
that  guidance.  Both  the  speed  of  the  C2  process  and 
the  distribution  of  C2  work  across  command  centres 
(particularly  planning  and  operations  management)  will 
vary  greatly  depending  on  the  commander’s  style  on 
this  dimension.  National  doctrine  and  practice  may  also 
influence  this  factor. 

Other  Typologies 

Other  typologies  of  command  styles  are,  of  course, 
possible  and  may  be  more  relevant  to  particular  C2 
analyses.  Human  behaviour  experts  (e.g.,  cognitive 
and  organisational  psychologists  and  anthropologists) 
should  be  recruited  to  the  project  team  if  novel 
categories  are  developed.  However,  the  most 
important  issue  when  dealing  with  command  style  is 
whether  it  is  included  in  the  analysis  at  all.  The  C2 
research  related  hypotheses  under  analysis  should 
dictate  the  forms  of  command  style  examined. 
However,  the  impact  of  command  style  should  only 
be  examined  because  it  appears  to  be  necessary  to 
answer  the  analytic  question(s)  of  interest.  Otherwise 
it  tends  to  introduce  a  level  of  complexity  that  may 
confound  the  other  analyses  underway. 
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Organisational  Factors 

All  elements  of  the  C2  system  are  ultimately  related  to 
one  another.  The  linkage  between  human  and 
organisational  issues,  however,  is  particularly  direct 
and  close.  Properly  done,  organisational  design 
reflects  the  interaction  among  the  tasks  to  be  done, 
the  people  available  to  perform  them,  and  the  systems 
or  tools  that  support  those  people.  Hence,  the  “proper” 
organisation  of  C2  depends  in  large  measure  on  the 
capabilities,  training,  and  experience  of  the  people  in 
the  C2  system. 

Organisation  is  a  serious  subject  in  military  analyses. 
For  centuries  the  military  has  sought  to  implement 
unambiguous  relationships  and  responsibilities.  Unity 
of  command  is  a  central  principle  of  war.  When  it  has 
been  lost  or  comes  into  question,  as  in  OOTW,  the 
professional  militaries  of  the  world  have  found 
themselves  very  uncomfortable.  Fortunately,  military 
organisational  issues  are  driven  by  a  fairly  small  and 
finite  list  of  principles.  Assessment  teams  asked  to 
work  on  C2  issues  can  use  the  known  list  of  factors  as 
a  checklist  about  organisational  differences  to 
determine  whether  they  need  to  build  organisational 
matters  into  their  research  designs.  This  includes  the 
issue  of  informal  relationships  that  may  have  evolved 
in  order  to  overcome  organisational  deficits  and  thus 
streamline  day-to-day  operations.  In  fact,  an 
organisational  design  and  command  style  that  are 
supportive  of  building  informal  relationships  may 
provide  the  flexibility  for  efficiently  handling  the 
manifold  demands  facing  commanders  in  OOTW. 
Also,  it  should  be  recognised  that  organisational 
implications  which  are  perceived  as  detrimental  to  the 
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interests  of  the  affected  individuals  and  groups 
inevitably  would  jeopardise  cooperation,  technical  and 
procedural  improvements  in  C2  notwithstanding. 

Organisational  Differences 

The  principal  differences  between  military 
organisations  are  related  to  structure,  function,  and 
capacity.  Any  change  or  innovation  that  can  be 
introduced  in  a  C2  organisation  falls  into  one  of  these 
three  categories  which,  therefore,  may  be  used  to 
guide  analysts  when  structuring  a  problem. 

Structural  differences  include: 

•  The  number  of  echelons  or  layers  in  the 
command  structure; 

•  The  span  of  control  for  nodes  in  the 
command  structure; 

•  The  pattern  of  linkages  between  those  nodes 
(e.g.,  hierarchical,  spokes  of  a  wheel,  multi- 
connected,  networked); 

•  Permanent  versus  transitory  organisational 
relationship;  and 

•  Formal  versus  informal  relationships. 
Functional  differences  include: 

•  The  distribution  of  responsibility:  where  functional 
activities  are  located  (e.g.,  intelligence,  logistics, 
command,  civil  military  cooperation  [CIMIC]); 

•  The  distribution  of  authority  (ideally  co¬ 
located  with  responsibility); 
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•  The  distribution  of  information; 

•  Functional  specificity  (e.g.,  fire  support  vs. 
infantry  or  close  air  support  vs.  defensive 
counter  air)  vs.  general  and  integrated 
military  capabilities  (mission  tailored  task 
forces);  and 

•  Degree  of  ambiguity  in  command  relationships. 
Difference  in  capacity  is  related  to  differences  in: 

•  Personnel  (e.g.,  quality,  training,  experience); 

•  Communications  systems  and  architectures; 

•  Information  processing  systems  and 
architectures;  and 

•  Field  training  and  operational  experience. 

All  these  dimensions  can  be  modelled,  some  more 
easily  than  others.  However,  the  assessment  team’s 
challenge  is  to  identify  those  organisational  factors  that 
are  relevant  to  the  C2  analyses  underway.  This  issue 
must  also  be  addressed  knowing  that  organisational 
factors  are  interrelated:  changing  one  may  change 
others.  For  example,  the  decision  to  eliminate  a  level 
of  hierarchy  within  a  military  organisation  may  have  a 
profound  influence  on  the  span  of  control.  Similarly, 
changing  the  distribution  of  information  so  that  it  no 
longer  follows  the  chain  of  command  may  have 
profound  implications  for  the  ambiguity  of  command 
relationships.  Similar  effects  can  be  expected  when 
coalition  operations  involve  ad  hoc  members. 
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Treatment  of  Organisational  Factors 

Because  of  the  large  numbers  of  organisational 
variables  that  may  be  relevant  to  the  analysis  of  C2 
issues,  they  must  be  approached  carefully  and 
systematically.  When  possible,  organisation  theory 
expertise  should  be  brought  into  the  assessment  team. 
Review  of  organisational  issues  is  treated  in  a  two- 
step  process  guided  by  a  hypothesis  testing  logic.  The 
first  step  should  assess  whether  any  organisational 
variable  is  being  manipulated  directly.  For  example,  a 
decision  to  move  from  warfare  domain  task  forces1  to 
mission  tailored  task  forces  with  air  and  land  units 
planning  and  operating  together  under  a  joint 
commander  would  be  seen  as  a  direct  manipulation 
of  organisational  factors  and  should  be  studied  as 
such.  The  second  step,  a  search  for  indirect  effects  of 
organisational  factors,  may  be  more  difficult  and  will 
require  that  the  assessment  team  use  the  list  of 
possible  factors  as  a  checklist  and  think  through 
whether  they  may  be  altered  in  a  prepositional  (if,  then) 
logic.  An  assessment  team  that  posits  a  relationship 
between  the  C2  analysis  and  an  organisational  issue 
should  be  able  to  make  a  clear  statement  of  the 
hypothesis  and  the  causality  anticipated.  This  will 
enable  the  research  design  to  cover  not  only  the  gross 
effect  anticipated,  but  also  the  underlying  causal 
mechanism(s)  that  will  be  present  if  the  proposition  is 
correct.  Adopting  this  hypothesis  approach  is  also  a 
safeguard  against  assuming  that  organisational  issues 
are  easily  or  well  understood  and  can  be  treated  by 
assumption.  In  fact,  the  organisational  arena,  like  that 
of  human  factors,  is  one  of  the  most  difficult  in  C2 
analysis  and  must  be  approached  with  care  and  rigour. 
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For  example,  the  small  group  literature  makes  a  clear 
prediction  that  multi-connected  groups  will  be  able  to 
generate  better  answers  to  complex  problems,  but  will 
take  more  time  to  do  so  than  either  hierarchies  or  star 
shaped  groups.  The  causal  mechanism  in  that  theory  is 
greater  dialogue  and  the  representation  of  more 
independent  viewpoints.  Moreover,  these  richer 
discussions  are  expected  to  take  more  time.  All  other 
things  being  equal,  multi-connected  groups  that  are  found 
to  generate  better  answers  to  complex  problems  should 
also  engage  in  more  dialogue  and  be  found  to  have 
considered  more  information  and  or  solutions.  Modellers 
who  want  to  take  advantage  of  this  factor  to  explore 
alternatives  to  traditional  hierarchical  military 
decisionmaking  must  also  include  the  negative  features 
in  their  C2  models  (e.g.,  demands  for  more  time  from 
already  overburdened  staffs  and  slower  decisionmaking). 

Roles 

The  concept  of  a  role  comes  from  sociology.  A  role  is 
a  set  of  behaviours  expected  by  the  self  and  others. 
For  most  military  systems  the  roles  of  commander  and 
key  staff  are  well  understood  and  arise  from  a 
combination  of  tradition,  training,  experience,  and 
rational  planning.  Because  of  their  origin,  roles  are 
often  a  convenient  way  of  capturing  the  doctrine  about 
responsibilities  within  the  C2  system. 

Roles  can  be  used  to  capture  “syndromes”  or  sets  of 
related  attributes  within  a  C2  system.  For  example, 
an  object  oriented  program  might  have  different 
functional  organisations  and  their  leaders  might  be 
defined  as  having  different  attributes  that  reflect  their 
decisionmaking  responsibilities  and  the  information 
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they  would  receive  or  be  able  to  obtain  from  the 
information  network.  When  assessing  new  C2 
systems,  analysts  will  often  need  to  search  for  potential 
role  gaps  or  role  overlaps.  Either  of  these  would  be 
dysfunctional  in  military  operations  over  time.  Changes 
in  information  structures  also  have  considerable 
potential  for  creating  problems  of  this  type. 

Role  gaps,  role  overlaps,  and  even  role  conflicts  may 
be  more  of  a  problem  in  OOTW  because  military 
organizations  have  to  assume  new  roles  not 
embedded  in  their  tradition  and  experience  and 
requiring  them  to  cooperate  with  a  variety  of  non¬ 
military  actors  and  organizations  pursuing  their 
particular  objectives  and  the  roles  of  which  are  ill 
defined  in  many  cases.  Thus,  rather  than  being  a 
source  of  friction,  changing  or  adapting  existing 
information  structures  may  be  part  of  the  decision 
problems  that  the  C2  system  must  address  in  order  to 
eventually  bring  about  a  secure  and  stable  situation  in 
the  theatre  of  operations  through  overarching  “unity 
of  command.” 

Human  and  Organisational  Issues 
and  Technology 

The  foregoing  discussion  of  human  and  organizational 
issues  revealed  that  human  performance  and  behaviour 
as  well  as  the  organizational  design  of  C2  systems, 
and  therefore  the  effectiveness  of  C2,  depend  on  the 
available  communication  and  information  technology. 
In  fact,  a  C2  organization  resembles  a  system 
composed  of  interacting  human,  organizational,  and 
technological  elements  as  depicted  schematically  in 
Figure  6.1  adapted  from  Mandeles  et  al.  (1 996). 
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Figure  6.1  is  meant  to  illustrate  that  the  character  and 
performance  of  a  C2  system  may  change  as  anyone  of 
elements  in  these  three  categories  changes.  Moreover, 
since  the  human,  organizational  and  technological 
elements  are  closely  linked  in  most  cases,  optimising 
each  one  of  them  at  a  time  under  ceteris  paribus 
assumptions  for  the  other  two  rarely  ever  results  in  an 
efficient  C2  system  (Schot  and  Rip,  1996). 


Figure  6.1.  Representation  of  the  C2  System,  based 
on  Mandeles  et  al.  (1996) 

In  particular,  the  assessment  of  the  human-technology 
relationship  is  a  critical  requirement  that  implies 
challenges  that  can  be  both  social  and  technical  in  nature. 
Without  adapting  human  thought  and  behaviour  patterns 
and  organizational  structures  it  may  be  impossible  to 
exploit  the  potential  of  new  technology.  On  the  other  hand, 
the  performance  of  new  communication  and  information 
technologies  may  exceed  human  capabilities  of 
processing  information  (information  overflow)  and  thus 
result  in  a  degradation  of  human  performance  and  overall 
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effectiveness  of  a  C2  system,  the  improvement  of 
technical  parameters  notwithstanding. 

The  challenges  of  adapting  technological  capabilities 
to  meet  human  capabilities  and  the  requirements  of  the 
social  interaction  processes  of  commanders  and  staff, 
and  non-military  actors  and  populations  in  OOTW, 
require  socio-technical  assessment  approaches  of  the 
kind  that  evolved  in  the  fields  of  science  and  technology 
studies  (STS),  technology  assessment  (TA)  (Rip. 1 995), 
and  constructive  technology  assessment  (CTA)  (Van 
de  Poel,  1999). 

Integrated  Analyses 

Because  the  issues  arising  from  human  and 
organisational  factors  are  so  complex  and  so  tightly 
coupled,  C2  assessment  teams  often  use  integrating 
tools  to  define  the  key  dimensions  relevant  to  their 
analyses  and  explore  the  relationships  between  and 
among  them.  Integrating  tools  are  those  that  use 
selected  key  factors  with  powerful  influence  to  cut 
through  the  clutter  and  detail  implied  by  trying  to  study 
everything  and  concentrate  instead  on  the  most 
important  elements  in  the  problem.  These  key  driving 
factors  are  used  to  conduct  a  simpler  analysis  that 
can  then  be  augmented  by  sensitivity  analyses  and 
analytic  excursions  to  ensure  that  the  problem  has 
been  fully  and  properly  understood. 

For  example,  Figure  6.2  has  been  used  to  illustrate 
the  relationship  between  the  time  available  to  make  a 
decision,  the  complexity  of  the  decision,  and  the 
uncertainty  of  the  information  available  about  the 
situation.  These  three  factors  also  reflect  the  risk  or 
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opportunity  inherent  in  a  military  situation.  The  more 
complex  a  situation,  the  less  time  available,  and  the 
greater  the  uncertainty  of  the  available  information, 
the  greater  the  risks  (and  opportunities)  present. 

While  each  of  these  three  dimensions  can  be  examined 
independently,  considerable  insight  can  be  derived 
from  examining  them  as  a  related  set.  This  examination 
normally  begins  as  an  exercise  in  hypothesis 
generation,  but  can,  as  research  is  accomplished,  be 
converted  into  a  component  of  a  knowledge  base.  That 
is,  as  evidence  confirming  or  calling  for  revising  the 
key  hypotheses  is  generated,  the  graphic  becomes  a 
way  of  conveying  known  relationships  and  generating 
new  propositions  about  regions  or  subspaces  that  have 
not  yet  been  examined  empirically. 

In  some  sense,  one  corner  of  this  cube  represents  the 
worst  of  all  C2  worlds — almost  no  time  available,  an 
enormously  complex  problem,  and  considerable 
uncertainty  about  the  situation.  Past  research  suggests 
that  when  these  conditions  exist  the  decisionmaker  has 
no  choice  except  to  use  “best  professional  judgement” 
to  match  the  operational  situation  to  some  class  of  well- 
understood  military  situations  and  act  accordingly. 
(Hayes,  1994).  However,  decisionmaking  theory  also 
indicates  that  the  wise  commander  will  take  short-term 
actions  designed  to  create  more  time  and/or  more 
information  and  thereby  relocate  the  problem  to  a 
“better”  portion  of  the  space.  A  “risk  averse”  commander 
will  clearly  attempt  this  transformation  of  the  situation. 
However,  a  more  risk  oriented  leader  may  attempt  to 
cut  through  the  fog  of  war  with  decisive  action. 


Time  Available  I 

Figure  6.2.  Decisionmaking  Drivers 

The  opposite  corner  of  this  analytic  space,  defined  as 
ample  decision  time  available,  limited  complexity,  and 
low  uncertainty,  provides  the  ideal  situation  for 
decomposition  of  the  problem  and  development  of 
“optimal”  military  plans.  Many  innovations  in  C2 
systems  are  designed  to  move  the  situations  facing 
commanders  of  friendly  forces  toward  this  region. 
Indeed,  Van  Creveld’s  analysis  of  C2  defines  it  as  a 
search  for  greater  certainty  (Van  Creveld,  1985). 

This  cube  also  emphasises  an  imperfectly  understood 
dimension  of  C2  systems  and  the  decisionmaking  they 
imply.  That  dimension  is  the  speed  at  which  the  situation 
is  changing  (the  pace  of  operations)  in  relation  to  the 
time  required  to  make  and  implement  a  military  decision 
(the  speed  of  the  C2  system).  Where  the  speed  of  the 
C2  system  is  faster,  proactive  decisions  are  possible. 
When  the  pace  of  operations  is  faster,  decisions  must 
be  reactive.  The  commander  who  is  capable  of  making 
decisions  that  transform  the  operation  from  reactive  to 
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proactive  is  rare  and  enjoys  vision  not  only  about  what 
is,  but  also  about  what  is  possible. 

This  key  relationship  (pace  of  operations  to  speed  of 
the  C2  system)  is  the  driving  force  behind  the  observe, 
orient,  decide,  act  (OODA)  loop  and  the  resulting 
guidance  to  seek  to  “turn  inside  the  enemy’s  C2  loop.” 
However,  C2  analysts  must  constantly  discipline  their 
analyses  away  from  assuming  that  speed  alone  is  a 
desirable  attribute  of  a  C2  system  or  organisation. 
Making  and  implementing  bad  decisions  quickly  will 
result  in  more  rapid  failures,  not  military  success.  As 
is  discussed  in  detail  in  Chapter  5:  Measures  of  Merit, 
multiple  dimensions  of  performance  need  to  be 
analysed  whenever  C2  systems  are  assessed. 
However,  this  requirement  to  look  at  multiple 
dimensions  in  order  to  assess  C2  does  not  obviate 
the  value  of  performing  integrated  analyses  of  human 
factors  and  organisational  issues. 

Conclusions 

•  Issues  of  human  performance  and  behaviour 
should  be  incorporated  in  models  used  to 
analyse  issues  that  require  human  activity  either 
in  the  form  of  performance  parameters  or 
appropriate  sub-models  on  behaviour. 

•  Decisionmaking  that  is  rule  or  algorithmically 
based  can  be  modelled  directly,  but  error  rates 
should  be  estimated  if  humans  are  involved  in 
the  relevant  decisionmaking. 

•  Simple  decisions  are  programmable  (with 
appropriate  error  rates),  but  also  require 
estimations  of  when  decision  would  be  made. 
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•  Complex  decisions  can  be  treated  with  “human  in 
the  loop”  tools  and  techniques,  but  new 
techniques  are  being  developed  and  applied 
(see  Chapter  5). 

•  Style  of  command  and  decisionmaking  should 
be  considered  in  C2  analyses  that  focus  on 
specific  decisionmaking. 

•  Organisational  issues  can  be  decomposed  into 
constituent  elements  for  analysis. 

•  Hypotheses  or  propositional  structures  are  often 
the  most  useful  approach  to  human  factors  and 
organisational  issues. 

•  Integrated  analyses  involving  roles  or  selected 
aspects  of  a  problem  space  often  provide  a 
cohesive  approach  to  the  complexity  inherent  in 
human  factors  and  organisational  issues. 

•  Research  in  organisations  and  human  factors  is 
expanding  and  analysts  are  advised  to  consult 
the  available  literature.  Experts  in  this  area 
should  be  included  on  the  interdisciplinary  C2 
assessment  teams. 

•  Operational  knowledge  of  human  issues  is  still 
weak  in  many  areas.  Systematic  effort  is 
required  for  organising  a  consistent  program  for 
experiments  on  human  issues. 

Recommendations 

•  Human  and  organisational  issues  are  not  closed 
topics  and  should  be  considered  early  in  the 
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process  of  C2  analysis  when  the  problem  is 
formulated  and  a  strategy  is  adopted. 

•  Test  the  impact  of  established  decisionmaking  rules 
that  reflect  existing  guidance  as  part  of  an  analysis. 
This  is  an  indispensable  prerequisite  for  models. 

•  The  assessment  team  should  explore  the  quality 
of  data,  information,  or  knowledge  used  to  drive 
the  automatable  decisionmaking  process. 

•  Early  on,  the  assessment  team  should  establish 
working  relationships  with  the  potential  subjects 
of  the  study  but  be  careful  not  to  allow  this  to 
introduce  a  bias. 

•  Human  factors  and  organisational  expertise 
should  be  included  in  all  C2  assessment  teams; 
at  least  until  a  decision  can  be  made  that  they 
are  not  major  elements  of  the  analysis. 

•  Separate  human  performance  issues  (e.g., 
stress  and  fatigue)  from  cognitive  issues  (e.g., 
decisionmaking)  when  possible,  but  recognise 
that  they  interact. 

•  Use  a  checklist  (Annex  D)  and  hypothesis-testing 
logic  for  reviewing  human  and  organisational 
issues.  Remember  that  human  and  organisational 
issues  may  interact  as  do  the  structural, 
functional,  and  capacity  arenas  of  organisations. 

•  Integrated  analytic  tools  that  focus  on  key 
variables  that  drive  human  factors  and 
organisational  issues  will  often  prove  useful  in 
simplifying  analysis. 
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•  Sensitivity  analyses  are  particularly  important 
when  working  with  human  factors  and 
organisational  issues. 

•  Experiments  for  testing  hypotheses  on  human 
behaviour  underlying  the  C2  analysis  are 
strongly  recommended. 

Chapter  6  Acronyms 

ATS  -  Norman’s  Activation  Trigger  Scheme 

C2  -  Command  and  Control 

CIMIC  -  Civil-Military  Cooperation 

CISS  -  Center  for  Information  Systems  Security 

CTA-  Constructive  Technology  Assessment 

DISA-  Defence  Information  Systems  Agency 

DSTL- Defence  Science  and  Technology  Laboratories 

MoM  -  Measures  of  Merit 

OA  -  Operations  Assessment 

OODA  loop -Observe,  Orient,  Decide,  Act 

OOTW  -  Operations  Other  Than  War 

OR/OA- Operations  Research/Operations  Assessment 

STS-  Science  and  Technology  Studies 

TA- Technology  Assessment 

TTP- Tactics,  Techniques,  and  Procedures 

VTC- Video  Teleconference 
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'Land  warfare  with  one  commander,  air  warfare  with  another, 
each  reporting  directly  to  a  joint  commander. 


CHAPTER  7 


Scenarios 


Before  beginning  operations  you  must, 
without  any  indulgence  or  self- 
deception,  examine  objectively  every 
step  that  the  enemy  might  undertake 
to  thwart  your  plan,  and  consider  in 
each  conceivable  case  what  means 
are  open  to  you  to  fulfill  your  goal.  The 
more  you  anticipate  the  difficulties  in 
advance,  the  less  surprised  you  will  be 
should  you  encounter  them  during  the 
campaign.  Besides,  you  have  already 
thought  about  these  obstacles 
deliberately,  and  with  composure  you 
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have  perceived  the  means  of  avoiding 
them,  so  nothing  can  surprise  you. 

— Frederick  the  Great 

Plans  that  assume  the  likelihood  of 
one  particular  world  run  the  risk  of 
being  seriously  wrong. 

— James  A.  Dewar,  Carl  H.  Builder,  William  M.  Hix, 
Morlie  H.  Levin  from  Assumption-Based  Planning — 
A  Planning  Tool  for  Very  Uncertain  Times 

Purpose  of  Scenarios 

The  report  of  the  NATO  Panel  7  Ad  Hoc  Working 
Group  on  the  Impact  of  C3I  on  the  Battlefield  gave 
extensive  consideration  to  the  role  of  scenarios  in 
operations  analysis  (OA).  This  chapter  builds  on  that 
material  to  discuss  the  role  of  scenarios  in  Command 
and  Control  (C2)  analysis.  Figure  2.2  in  Chapter  2 
shows  the  role  played  by  scenarios  in  the  overall  C2 
assessment  process.  The  analysts  craft  a  set  of 
scenarios  to  provide  the  context  or  environment  for 
the  conduct  of  the  operational  analysis.  The  scenarios 
bound  the  arena  of  the  analysis  and  are  used  by  the 
analyst  to  focus  the  analysis  on  central  issues. 

Definitions 

These  definitions  are  of  particular  relevance  to  this 
Code  of  Best  Practice  (COBP): 

Scenario 

A  description  of  the  area,  the  environment,  means, 
objectives,  and  events  related  to  a  conflict  or  a  crisis 
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during  a  specified  time  frame  suited  for  satisfactory 
study  objectives  and  the  problem  analysis  directives. 

As  described  in  Chapter  4,  scenarios  consists  of  four 
elements — a  context  (e.g.,  a  characterisation  of  a 
geopolitical  situation),  the  participants  (e.g.,  intentions, 
capabilities  of  blue,  red,  others),  the  environment  (e.g., 
natural — weather  and  manmade — mines),  and  the 
evolution  of  events  in  time.  In  C2  assessments,  the 
purpose  of  scenarios  is  to  ensure  that  the  analysis  is 
informed  by  the  appropriate  range  of  opportunities  to 
observe  the  relevant  variables  and  their  interrelationships. 

Approved  Scenario 

In  order  to  support  the  analysis,  the  use  of  an  approved 
scenario  is  more  relevant  than  to  analyse  a  current 
situation  or  future  plans.  The  impact  of  all  the  elements 
is  simpler  to  analyse. 

In  some  countries  these  scenarios  are  mainly 
developed  by  a  Strategic  Committee  and  are  designed 
to  meet  the  government  strategy.  The  first  step  is  to 
implement  a  steering  group,  composed  of 
representatives  of  the  government  and  members  of 
the  Armed  Forces,  which  has  the  responsibility  to 
describe  all  the  commitments  or  the  precise  regions 
in  which  any  major  event  may  have  an  impact  on  the 
foreign  politic  or  on  the  economy.  This  basic  work  is 
then  forwarded  to  a  larger  group  or  a  next  military  level 
which  shares  the  responsibility  for  implementing  the 
main  recommendations  in  a  more  strategic  field.  These 
scenarios  specify  the  elements  that  must  be  kept  in 
mind  in  the  generic  planning  process  or  in  the 
operational  studies  in  relation  with  the  use  of  military 
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forces.  These  scenarios  also  describe  the  global  threat, 
the  geographic  areas,  the  political  and  military 
objectives  and  the  level  of  force  commitment. 

With  this  basic  approach  done,  the  analyst  or  his 
military  counterpart  has  only  to  adapt  those  guidelines, 
to  compare  his  study  to  the  scenarios  and  to  propose 
the  more  relevant  alterations  to  his  sponsor  and  his 
assessment  team  {See  Assessment  Team  paragraph 
in  Chapter  2). 

At  this  stage  the  decision  to  keep  this  scenario,  to 
diverge  or  to  choose  another  one  will  be  discussed 
among  the  stakeholders.  The  products  of  this  process 
are  called  approved  scenarios. 

After  that,  at  any  time  the  analysts  may  develop 
vignettes  using  the  approved  scenarios  as  a  base  in 
order  to  focus  on  a  specific  issue.  These  vignettes 
can  be  used  as  small  scenarios  to  explore  a  particular 
topic.  The  use  of  prepared  scenarios  must  not  be 
considered  as  a  limitation  for  customers  or  analysts 
because  all  the  data  could  be  adapted  by  the  customer 
or  the  assessment  team,  the  only  limitation  is  to  avoid 
the  re-use  of  this  particular  scenario  for  a  study  not  in 
close  relation  with  the  original  one. 

Planning  Scenario 

A  planning  scenario  is  one  in  which  these  elements 
are  defined: 

•  Time  frame  of  the  analysis; 

•  Geographic  area; 

•  Meteorological  environment; 


Chapter  1  1 67 


•  Political,  historical,  economic,  and  social  context; 

•  Mission  objectives  and  constraints; 

•  Level  of  threat; 

•  Friendly  forces  (and  links  between  each 
members  of  the  coalition); 

•  Adversary  forces  (e.g.,  enemy  order  of  battle); 

•  Neutral  or  uncommitted  forces; 

•  Non-combatant  (non-government  organisations 
[NGOs],  international  organisations,  etc.)  and 
other  relevant  actors;  and 

•  Media  (when  relevant). 

Operational  Scenario 

An  operational  scenario  contains  additional  details 
especially  with  respect  to  threats,  orders  of  battle, 
tactics,  rules  of  engagement,  courses  of  action, 
deployment,  end  state  and  reserves. 

Vignette 

The  term  “vignette”  is  sometimes  used  for  a  scenario 
that  is  not  approved.  The  term  is  also  primarily  used 
for  smaller  scenarios,  particularly  as  excursions  from 
the  main  scenario. 

Role  of  Scenarios  in  C2  Analysis 

In  general,  the  ideal  OA  is  scenario  independent.  All 
relevant  factors  can  be  identified  and  dealt  with 


168  NATO  Code  of  Best  Practice  for  C2  Assessment 


empirically  and  algorithmically  across  a  range  of  military 
contexts.  However,  C2  involves  human  behaviour, 
organisations,  missions,  and  other  complex 
phenomena.  Human  behaviour  is  very  difficult  to  put 
into  equations  (see  “Human  Decisionmaking”  in  Chapter 
6).  There  is  no  single  linear  dimension  for  organisations 
or  human  issues.  Moreover,  military  missions  do  not 
form  simple  dimensions.  Therefore,  for  most  C2 
analyses,  the  context  must  be  defined.  This  is  the  role 
for  which  the  analyst  defines  the  scenarios. 

The  formulation  of  the  original  problem  dictates  the 
contents  of  the  scenarios.  There  are  no  overall 
scenarios  that  are  independent  of  a  specific  problem. 
Scenarios  are  never  truly  generic,  but  rather  are 
customised  if  only  by  the  assumptions  built  into  them. 
The  boundaries  of  the  scenario  space  should  be 
defined  in  part  by  the  issues  unique  to  the  problem 
under  analysis. 

Organisational  issues  include  the  involvement  of 
various  levels  of  military  and  non-military  hierarchies, 
including  different  command  levels.  This  requires  that 
scenarios  accommodate  analysis  across  different 
echelons  of  command.  Information  processing  and  the 
characteristics  of  information  must  also  be 
accommodated.  Human  factors  include  the 
decisionmaking  process  and  supporting  staff  activities. 

The  analyst  will  need  to  design  or  select  scenarios  to 
address  C2  under  a  broad  range  of  circumstances. 
This  taxonomy  of  C2  analysis  might  include: 

•  Defense-planning; 

•  Force  structure  and  organisation; 
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•  Mission  analysis; 

•  Doctrine/tactics  development; 

•  Cost-benefit/effectiveness  analysis; 

•  Training  and  education; 

•  Balancing  C2  systems  and  weapon/sensor  systems; 

•  C2  system  procurement,  which  will  often  require 
more  detailed,  task  specific  scenarios  to  cover 
the  range  of  relevant  system  uses;  and 

•  Non-combatant  actor’s  access  to  C2  systems 
and  integration  into  the  information  flow,  typically 
through  liaison  officers,  but  increasingly  through 
a  variety  of  other  means. 

In  essence,  the  role  of  a  scenario  is  to  define  a  set  of 
conditions  and  restrictions  to  enable  “good”  analysis 
as  well  as  to  create  a  structure  within  which  the  results 
of  the  analysis  can  be  understood  and  interpreted. 

Understanding  and  Interpreting  the 
Results  of  Operational  Analysis 

The  analyst  uses  scenarios  to  understand  and  interpret 
the  value  of  OA  study  results  for  the  focus  of  the 
analysis.  The  scenarios  provide  the  context  in  which 
the  C2  system  will  be  assessed.  It  should  reflect  the 
scenarios  envisaged  by  the  originator  of  the 
requirement  for  the  system.  Often  artificial  constraints 
must  be  introduced  into  the  scenarios,  due  to  cost 
considerations,  to  properly  focus  the  analysis,  or  both. 
The  scenario  developer  must  have  an  appreciation  of 
the  objectives  of  the  simulation,  experiment,  or 
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exercise  analysis  plan  in  order  to  determine  the  artificial 
constraints  necessary  to  facilitate  the  analysis.  The 
analyst  needs  to  be  aware  of  scenario  assumptions 
and  artificial  constraints. 

There  are  several  essential  questions  in  C2  analysis 
that  should  be  addressed  in  the  scenario  considerations: 

•  operational  benefits  of  C2  to  be  translated  in 
the  definitions  of  Measures  of  Policy 
Effectiveness  (MoPE)  and  Measures  of  Force 
Effectiveness  (MoFE); 

•  required  or  desired  performance  thresholds  and 
nominal  Measures  of  Merit  (MoM)  values;  and 

•  the  impact  of  an  improved  volume,  accuracy,  and / 
or  quality  of  information  on  the  final  outcome. 

Developing  and  Specifying  Scenarios 

Prerequisites  in  Scenario  Definition 

Several  prerequisites  are  essential  before  using 
scenarios  for  C2  analysis: 

•  Approval:  the  analyst  should  strive  for  the 
creation  of  a  family  of  approved  scenarios.  In 
creating  a  family  of  approved  scenarios,  which 
reflect  the  mission  objectives  and  force 
capabilities  and  cover  all  significant  warfare 
areas,  the  analyst  facilitates  the  scenario 
development  process  to  a  great  extent,  because 
references  to  basic  assumptions  and  conditions 
can  be  made.  This  will  also  increase  the  validity 
of  the  analysis  in  the  eyes  of  the  client  and 
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facilitate  comparison  of  the  results  from  different 
studies  and  analyses  that  use  the  same 
approved  scenarios; 

•  Breadth:  a  scenario  should  reflect  those  factors 
that  are  hypothesised  to  have  a  significant 
impact  on  C2  issues; 

•  Capability:  a  scenario  should  stress  C2 
capabilities,  including  human  and  organisational 
factors  (military  and/or  civilian)  where 
appropriate;  and 

•  Credibility:  scenarios  should  include  logical 
assumptions  about  the  environment  under  analysis. 

Scenarios  should  represent  plausible  real  world 
situations.  The  synthetic  scenario  environment  should 
be  consistent  across  OA  studies.  The  scenarios  will 
gain  credibility  if  a  broadly  based  scenario  team  is 
involved  in  the  process  from  the  outset.  This  team 
should  include  a  variety  of  perspectives  and  expertise, 
such  as: 

•  OA  analysts  for  defining  the  required  scenario 
information,  to  avoid  biasing  the  analysis  by 
selecting  an  inappropriate  scenario  space,  and 
for  working  out  the  process  of  framing  of  the 
scenario  space; 

•  Defence  concept  planners  to  propose  options 
and  highlight  the  critical  factors; 

•  Policy  makers  to  ensure  that  the  strategic 
decision  points  and  the  alternative  options  for 
each  of  these  decision  points  are  clear  and 
consistent  with  Defence  Policy;  and 
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•  Subject  matter  experts  to  credibly  explore  the  range  of 
possibilities  (scenario  space)  and  foster  discussions. 

C2  Organisation  Infrastructure  and 
Operating  Environment 

Organisation  infrastructure  and  environment  are  often 
pre-set  conditions  and  not  the  subject  of  the  study.  They 
include  C2  concepts  of  operation,  decision  hierarchy  of 
the  units  under  consideration,  degree  of  technological 
competence  relative  to  that  of  the  adversary, 
requirements  or  objectives  placed  upon  the  system  in 
terms  of  speed,  accuracy,  flexibility,  etc.,  and  the 
impacts  of  terrain,  weather,  and  adversary  activities. 

C2  Processes 

The  scenarios  need  to  provide  for  the  realistic 
execution  of  the  C2  processes.  These  include  the  span 
of  control  of  the  various  military  command  levels  and 
civilian  authorities,  information  management  schemes, 
information  flows,  the  elements  of  the  decision  cycle, 
the  decision  processes  (course  of  action  development, 
planning,  directing),  and  the  communications 
processes  and  capabilities  (data  update  rates, 
throughput,  reliability,  accuracy,  etc.).  All  too  often 
these  issues  are  characterised  by  simple  performance 
indicators  and  not  examined  in  detail.  They  may  also 
be  an  important  subject  of  the  study.  These  factors 
need  to  be  explicitly  built  into  the  scenarios. 

C2  Systems 

Characteristics  of  C2  systems  are  directly  related  to 
system  improvements.  They  include  system 
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performance  parameters,  command  and  control 
information  systems  (CCIS),  data  availability, 
intelligence  functions  (fusion,  correlation,  aggregation, 
etc.),  surveillance,  targeting  and  acquisition  (STA), 
communications  systems,  throughput,  and  so  forth. 

Human  Factors 

C2  studies  are  complex  in  nature.  One  of  the 
complicating  factors  is  the  involvement  of  human 
beings  and  their  interpretation  of  a  situation,  order,  or 
rule  of  engagement.  These  factors  can  be  covered  by 
the  “aggregation/de-aggregation”  phenomena  in  the 
command  chain.  Human  factors  have  to  be  included 
in  the  analysis  and  modelling  activities,  but  guidelines 
on  how  to  integrate  the  “human  in  the  loop”  are  partly 
defined  in  the  scenarios.  (Chapter  6 — Human 
Decisionmaking  discusses  these  in  some  detail.) 

Miscellaneous 

C2  studies  usually  cover  various  levels  of  hierarchy. 
However,  the  nature  of  C2  issues  does  not  materially 
change  for  the  various  command  levels.  The  analyst 
may  need  to  perform  a  cost/benefit  analysis  on  the 
inclusion  of  lower  level  C2  issues  in  a  closed  simulation 
model.  The  analyst  needs  to  consider  to  what  extent 
these  issues  (e.g.,  performance  of  a  logistic  information 
system)  need  to  be  converted  to  enabling  factors  (e.g., 
sustainability,  operational  delays)  at  the  higher  levels. 

These  issues  are  mentioned  to  illustrate  that  in  the  scenario 
definition  a  great  deal  of  attention  is  required  to  ensure 
that  the  scenario  enables  the  proper  C2  issue  to  be 
addressed  in  problem  definition.  The  elements  are  often 
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dependent  on  each  other.  For  example,  some 
shortcomings  in  C2  systems  can  be  compensated  for  by 
alterations  in  C2  processes.  Similarly,  inefficiencies  in  C2 
processes  can  be  met  by  an  adaptable  C2  organisation. 
The  relationships  between  these  issues  should  be 
recognised  and  taken  into  account  in  the  scenarios. 

Approach  to  Scenario  Development 

This  section  describes  a  framework  for  the  definition 
of  a  scenario.  Then,  based  on  this  framework,  some 
specific  aspects  of  actually  using  scenarios  for  C2 
analysis  are  addressed. 

Scenario  Structure 

The  general  scenario  framework  developed  from  the 
NATO  Panel  7  Ad  Hoc  Working  Group  on  the  Impact 
of  C3I  on  the  Battlefield  has  been  adopted  (Figure  7.1 ). 

At  the  first  two  levels,  a  description  of  the  external 
factors  and  the  capabilities  of  the  actors,  including 
national  security  interests;  the  political,  historical,  and 
military  situation;  and  the  acting  assumptions, 
boundary  conditions,  and  limitations  related  to 
adversaries,  threats,  risks,  coalition  partners,  warfare 
domains  etc.  are  given.  Very  often  a  reference  to  an 
approved  scenario  will  suffice. 

At  the  third  level,  the  mission  environment  is  defined. 
Whether  it  is  a  generic,  virtual  geographic  environment 
or  a  specific  geographical  area  is  not  important.  What  is 
essential  is  that  the  mission  environment  be  addressed. 
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Figure  7.1 .  The  Scenario  Framework 


The  intermediate  level  is  the  most  challenging  one.  The 
actual  military  problem  has  to  be  projected  on  the 
mission,  the  military  forces  and  capabilities,  the  civilian 
capabilities,  and  the  resources  available.  A  scenario 
must  be  developed  by  coherently  aggregating  a  number 
of  components  or  dimensions  with  their  attached  values, 
taking  into  account  of  the  problem  formulation.  It  should 
address  at  least  these  components: 

•  Geopolitical  situation,  including  historical  aspects; 

•  Geographical  area; 

•  Availability/usability  of  civil  infrastructure; 

•  Terrain  and  climate; 

•  Political/economical/military  objectives; 

•  Level  of  violence,  type  of  warfare  areas,  and 
preparation  times; 

•  Mission  context  and  objectives; 

•  Mission  tasks  and  goals; 
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•  National  contributions  and  roles  within  the 
coalition; 

•  Order  of  battle; 

•  Doctrines,  procedures,  (range  of  acceptable) 
rules  of  engagement,  and  concepts  of  operation; 

•  Temporal  factors  (e.g.,  anticipated  duration 
of  operations); 

•  Desired  end  states; 

•  Opposition/threat/risks; 

•  Adversary  forces  and  their  organisation; 

•  Other  actors  (neutral  and  uncommitted 
forces,  non-combatants,  refugees,  IOs, 
NGO’s,  Red  Cross,  etc.); 

•  Level  of  threat  and  risk; 

•  C2  structures; 

•  Interaction  between  friendly,  adversary,  and 
other  information  systems;  and 

•  Assumptions/hypotheses/axioms  about  level  of 
technology,  impact  on  information  defence,  etc. 

The  actual  military  problem  will  be  placed  in  the  context 
of  the  friendly  and  adversary  military  forces  involved,  e.g. : 

•  Force  organisation,  C2  structure,  force  components; 

•  Doctrines,  tactics,  rules  of  engagements; 

•  Courses  of  action; 

•  Information  systems;  and 
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•  Logistics. 

The  framing  of  scenarios  begins  with  the  identification  of 
the  key  dimensions  relative  to  the  problem  being 
addressed.  The  search  of  these  dimensions  is  delicate 
and  requires  thorough  reflection,  for  instance,  on  the 
results  of  a  structural  analysis.  Once  identified,  each  key 
dimension  or  factor  must  be  characterised  by  a  range  of 
possible  values  or  sectors,  which  enables  the  set  up  of 
the  sector-factor  matrix.  This  sector  factor  matrix  is  then 
analysed  through  a  morphological  analysis  (construction 
and  reduction  of  the  morphological  space  or  space  of 
“all  possible  scenarios”),  which  provides  the  user  with  a 
set  of  appropriate  scenarios.  The  scenarios  are 
dependent  upon  the  problem  and  the  objective  of  the 
study  and  can  range  from  generic  to  very  specific.  The 
analyst  will  select  the  level  of  detail  required  to  drive  and 
focus  the  model. 

Using  Scenarios  in  C2  Assessment 

It  has  already  been  noted  that  a  well-formulated  OA 
problem  definition,  guidelines  and  directives  for  how 
to  approach  the  analysis  should  accompany  a 
scenario.  Emphasis  in  this  COBP  is  given  to  C2 
elements  in  relation  to: 

•  Mission  scope:  as  stated  before,  one  of  the 
characteristics  of  C2  is  that  it  can  not  be  studied 
in  isolation.  C2  is  an  integrating  and  enabling 
factor.  As  a  consequence  there  is  a  tendency  to 
consider  the  entire  mission,  covering  all  the 
arenas  connected  to  it  (from  logistics  to 
manoeuvre,  from  artillery  support  to  close 
combat,  from  security/police  issues  to  refugee 
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management).  Therefore,  the  typical  mission 
scope  for  C2  analysis  will  be  broad; 

•  Levels  of  hierarchy:  the  C2  chain  is  not  limited  to 
a  special  hierarchical  level;  information  and 
command  flows  are  running  from  the  lowest 
levels  to  the  higher  echelons  and  vice  versa.  As 
a  consequence,  there  is  a  tendency  to  cover  a 
wide  set  of  hierarchy  levels  in  considering  a  C2 
problem.  Single  layer  analyses  do  not  represent 
the  dynamics  of  military  problems  adequately  to 
answer  most  C2  issues;  and 

•  Aggregation/disaggregation:  the  aggregation  of 
data  flows,  data  fusion  in  support  of  intelligence 
processes,  merging  C2  items  to  more  abstract 
levels,  etc.  is  difficult  but  manageable.  The 
process  of  deleting,  merging,  and  combining 
information  is  reasonably  well  understood.  The 
integration  of  soft  factors  (e.g.,  human  and 
organisational)  is  less  well  understood  and  makes 
the  problem  more  difficult  to  simulate,  study,  or 
analyse.  In  particular,  the  effects  of  interactions 
between  functional  groups  and  echelons  must  be 
considered  when  decomposition  is  undertaken  for 
analytic  purposes. 

Mission  Scope  versus  Levels  of  Hierarchy 

Experience  with  prior  modelling  efforts  indicates  that 
the  scenario  developer  needs  to  understand  the 
relationships  between  the  scope  of  the  mission,  the 
hierarchy  assigned  to  conduct  the  mission,  and  the 
level  of  detail  at  which  the  echelon  operates.  In  general, 
as  shown  in  Figure  7.2,  the  broader  the  mission  scope, 
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the  higher  the  echelon  required  to  conduct  the  mission. 
Also,  higher  command  levels  tend  to  use  information 
at  higher  levels  of  aggregation,  or  less  detail,  than  lower 
levels.  The  scenario  developer  should  attempt  to 
operate  in  the  shaded  area  of  the  diagram,  matching 
the  echelon  levels  to  the  proper  aggregation  level  and 
mission  scope. 
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Figure  7.2  Intersections  in  Hierarchy 


In  many  complex  problems,  the  analyst  is  required  to 
subdivide  the  problem  into  smaller  parts,  perhaps  even 
using  different  models  or  model  federations  to 
represent  different  levels.  The  scenario  developer 
should  work  in  conjunction  with  the  analyst  to  ensure 
that  the  hierarchical  intersections  and  interactions  (e.g., 
organisation  and  infrastructure,  C2  processes,  and  C2 
resources)  are  properly  represented,  that  the  models 
are  consistent,  and  their  inputs  and  outputs  are 
properly  linked.  Figure  7.3  graphically  depicts  this 
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relationship,  with  the  rectangles  X,  Y,  and  Z 
representing  different  models. 


Mission  scope; 


Figure  7.3  Segmentation  Hierarchy  Range 

Aggregation/De-Aggregation:  Non-Causality 
Between  C2  Issues  At  Different  Levels 

Analysis  of  C2  issues  generally  requires  assessing 
the  effects  of  events  and  actions  across  command 
levels  to  determine  causality  between  actions  at  one 
echelon  and  events  at  another.  For  example,  the 
analyst  will  want  to  know  if  the  capability  to  make  faster 
decisions  at  battalion  level  has  an  impact  at  brigade 
level  or  higher.  Some  C2  items  have  an  impact 
throughout  the  entire  hierarchical  range,  some  affect 
only  one  other  level,  and  some  are  purely  local,  i.e., 
level-specific.  The  scenario  developer  will  need  to  be 
aware  of  the  analyst’s  requirements  in  this  area  in  order 
to  design  the  proper  linkages  between  events. 


Chapter  7  181 


Additional  Aspects 

Additional  Areas  of  Consideration: 

•  Specification:  the  purpose  of  the  C2  analysis  will 
influence  the  kind  of  scenarios  to  be  used.  There 
are  no  universal  generic  scenarios  for  C2 
analysis.  Some  nations  have  “validated” 
scenarios  that  can  be  taken  off  the  shelf  and 
modified  to  support  particular  analyses; 

•  Merging  mission  operations  areas:  mission  areas 
for  the  various  levels  of  intensity  of  conflict  and 
various  level  of  civilians  involvement  are  not 
discrete,  and  it  may  be  necessary  to  include 
elements  of  more  than  one  type  into  a  scenario; 

•  Decomposition:  sometimes  it  might  be  useful  to 
decompose  a  scenario  into  two  or  more  detailed 
scenarios  that  each  deal  with  a  certain  subset  of 
C2  issues.  This  is  more  or  less  analogous  to  the 
decomposition  into  one  or  more  mission 
operations  areas.  It  may  be  necessary  to  add 
some  vignettes  to  look  in  greater  detail  such  that 
all  C2  issues  of  interests  are  covered  and  that 
the  whole  range  of  relevant  OOTW  is  explored; 

•  Adversary-friendly  interaction:  the  attention  given 
to  adversary  and  friendly  C2  processes  should 
be  balanced  in  those  cases  where  adversary 
activities  are  germane  to  the  problem,  especially 
if  counter  C2  (information  defence)  is  part  of  the 
analytic  focus; 

•  Assumptions  and  guidelines:  the  scenarios  are 
part  of  the  process  in  developing  an  analysis  (See 


182  NATO  Code  of  Best  Practice  for  C2  Assessment 


Figure  2.3  in  Chapter  2).  In  the  scenario  phase,  all 
scenario  assumptions,  guidelines,  and  boundaries 
for  the  study  should  be  revisited.  Each  study  is 
executed  within  the  framework  of  the  scenario, 
and  therefore  the  study  findings  are  only  valid 
within  the  limitations  of  the  various  assumptions 
and  artificial  constraints  of  the  scenario; 

•  Traceability:  analysts  should  understand  which 
scenario  assumptions  and/or  boundary 
conditions  are  driving  factors  in  the  analysis.  A 
detailed  description  of  past  use  of  scenarios 
should  be  maintained  on  a  national  level  in  order 
to  avoid  duplication.  Such  a  repository  should 
also  contain  Verification,  Validation,  and 
Accreditation  (VV&A)  information  on  the 
scenarios;  and 

•  Awareness:  create  awareness  of  the  robustness  of 
the  overall  conclusions  and  decisions  and  be 
aware  of  the  degrees  of  uncertainty  in  the  scenario. 

Conclusions 

•  Using  a  scenario  for  C2  analysis  is  only  one  part 
of  a  larger  analytical  methodology.  The  context 
provided  by  the  scenario  impacts  in  other  areas, 
and  the  scenario  in  turn  is  affected  by  those 
same  areas; 

•Six  prerequisites  should  be  in  place  before  using 
a  scenario  for  C2  analysis: 

•  It  should  be  approved  for  the  assessment; 

•  It  should  reflect  the  factors  that  have 
significant  impact  on  C2  needs; 
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•  It  should  stress  C2  issues; 

•  It  should  be  militarily  credible; 

•  It  should  be  credible  in  terms  of  civil- 
military  objectives; 

•  It  should  facilitate  the  design  process; 

•  At  least  three  C2  elements  should  be  reflected  in  a 
scenario  in  order  to  make  it  useful  for  C2  analysis: 

•  The  C2  organisation  and  infrastructure, 
including  human  issues; 

•  The  C2  processes; 

•  The  C2  systems; 

•  Scenario  guiding  directives  should  indicate  how 
the  scenario  has  been  used  in  a  hierarchy  of 
scenarios  (interpretation  of  input  and  output 
events,  etc.); 

•  The  actual  C2  analysis  problem  usually  will  be 
broader  in  scope  than  OA  will  allow.  Hence 
Scenario  Analysis  in  combination  with  military 
and  civilian  judgement  (including  lessons 
learned)  must  bridge  this  gap;  and 

•  Analysts  need  to  use  multiple  scenarios;  no 
single  scenario  is  sufficient. 

Recommendations 

Practice 

•  Organise  a  set  of  scenarios  and  vignettes  that 
allow  the  analysis  to  cover  or  sample  the 
interesting  problem  space  for  the  C2  analysis; 
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•  Create  a  (national)  base  of  approved  scenarios 
and  vignettes  reflecting  the  civil-military 
objectives  within  the  national  hierarchy  of 
operations  and  thus  the  required  spectrum  of 
military  missions  including  OOTW  capabilities; 

•  Explicitly  identify  and  describe  the  scenarios  prior 
to  the  execution  of  a  study.  However,  it  might  be 
necessary  to  revisit  the  scenario  definition  during 
the  conduct  of  the  study; 

•  Information  and  hypotheses  on  threats, 
adversary  forces,  and  non-combatants  should  be 
addressed  in  the  scenario; 

•  Explicitly  identify  the  C2  aspects  under 
consideration  within  the  problem  definition;  and 

•  During  the  analysis,  the  key  scenario  assumptions 
should  be  identified  and  documented. 

Challenges 

•  Standards  forjudging  the  applicability  and 
accreditation  of  (existing)  models  should  be 
developed;  and 

•  For  coalition  C2  assessments  the  scenarios  should 
be  developed  or  adapted  by  teams  including 
representatives  from  all  participating  nations. 

Chapter  7  Acronyms 

C2  -  Command  and  Control 

CCIS  -  Command  and  Control  Information  Systems 

COBP  -  Code  of  Best  Practice 
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IO  -  International  Organisation 

MoFE  -  Measures  of  Force  Effectiveness 

MoM  -  Measures  of  Merit 

MoPE  -  Measures  of  Policy  Effectiveness 

NGO  -  Non-Government  Organisation 

OA  -  Operational  Analysis 

STA-  Surveillance  Targeting  and  Acquisition 

W&A-  Verification,  Validation,  and  Accreditation 
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CHAPTER  8 


Methods  and  Tools 


The  reasonable  course  of  action  in  any 
use  of  arms  starts  with  calculation. 

Before  fighting,  first  assess  the  relative 
sagacity  of  the  military  leadership,  the 
relative  strength  of  the  enemy,  the  size 
of  the  armies,  the  lie  of  the  land,  and 
the  adequacy  of  provisions.  If  you  send 
troops  out  only  after  making  these 
calculations,  you  will  never  fail  to  win. 

— Liu  Ji,  1 31 0-1 375,  Lessons  of  War 

War  is  essentially  a  calculation  of 
probabilities. 

— Napoleon 
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If  the  only  tool  that  you  have  is  a 
hammer,  you  tend  to  see  every 
problem  as  a  nail. 

— attributed  to  a  Harvard  anthropologist 

The  purpose  of  this  chapter  is  to  consider  the  best 
methods  for  representing  Command  and  Control 
(C2)  systems,  processes,  organisations,  and  their 
interaction  in  order  to  support  assessments  of  C2  over 
the  full  spectrum  of  operations,  to  include  Operations 
Other  Than  War  (OOTW).  The  key  objective  is  to 
establish  an  ‘audit  trail’  from  data  or  information 
collected,  through  its  processing,  presentation, 
dissemination,  and  use  to  the  performance  of  C2 
processes  and  organisation  as  well  as  to  high-level 
measures  of  their  effects  on  battle  or  operations 
outcome. 

This  chapter  of  the  revised  COBP  extends  the  code  to 
cover  method  and  tool  considerations  regarding 
OOTW.  The  material  in  this  chapter  is  a  distillation  of 
the  best  approaches  and  ideas  being  considered  in 
current  NATO  research  for  representing  C2  across  the 
full  spectrum  of  operations  in  all  models/tools. 

Types  of  Methods  and  Tools 

This  chapter  covers  all  tools  (simulations  or  other 
quantitative  or  qualitative  techniques),  whether  used 
for  analysis,  training,  or  operational  purposes  that  can 
be  used  to  assess  C2  processes,  performance,  and 
effectiveness.  Available  methods  and  tools  can  be 
categorised  into  four  distinct  groups: 
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•  Data  collection/generation:  methods  and  tools 
used  to  either  collect  or  generate  subjective  or 
objective  data  for  subsequent  analysis  from  live, 
virtual,  or  constructive  sources,  whether  past, 
present,  or  future; 

•  Data  organisation/relationship:  methods  and 
tools  used  to  organise  data  in  some  logical  way, 
or  used  to  establish  relationships  between  data. 
These  methods  and  tools,  rather  than  providing 
a  mathematical  solution  to  problems,  tend  to  be 
more  qualitative,  subjective,  and  exploratory 
techniques  based  on  expert  opinion,  judgement, 
and  interaction,  whether  obtained  directly  or 
through  role  playing.  Although,  in  some  cases, 
these  tools/techniques  may  totally  solve  the 
problem  at  hand,  more  often  they  will  illuminate 
other  associated  or  sub-problems,  determine 
areas  for  further  analysis,  or  provide  expert  input 
“data”  for  more  quantitative  “solving”  tools; 

•  Solving:  methods  and  tools  which  have  been 
typically  associated  with  operations  research, 
business,  mathematics,  computer  science, 
information  science,  engineering,  or 
management  science  which  tend  to  be 
quantitative  in  nature  and  which  usually  consist 
of  techniques  providing  mathematically  derived 
solutions,  even  if  the  data  analysed  is  subjective 
in  nature;  and 

•  Support:  methods  and  tools  used  to  collect, 
organise,  store,  and  explore  typically  large  sets 
of  empirical  data. 
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Table  8.1  provides  some  recommended  methods  and 
tools  that  fall  into  each  category.  The  intent  here  is 
not  to  provide  an  exhaustive  list  or  to  debate  into  which 
category  a  specific  method  or  tool  should  be  included, 
but  to  give  the  range  of  methods  and  tools  available 
to  the  analyst  for  C2  assessment.  While  the  range  and 
scope  of  potential  methods  and  tools  is  broad,  clearly 
the  emphasis  of  this  chapter  of  the  COBP,  and  of  the 
analysis  community  at  large  over  recent  years,  is  on 
constructive  modelling  and  how  best  to  enhance, 
orchestrate,  and  apply  it  to  the  assessment  of  C2 
impacts  on  battle/operations  outcome. 
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Table  8.1 .  Some  Recommended  Tools  by  Category  Type 
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Issues 

The  NATO  analytic  community  has  had  many 
challenges  in  the  past  analysing  the  effectiveness  of 
C2  related  systems  and  determining  what  it  is  that  sets 
it  apart  from  other  types  of  operational  analysis.  The 
added  complexity  and  number  of  confounded  variables 
in  OOTW  make  analysis  of  these  operations  even 
more  challenging.  The  key  to  the  problem,  no  matter 
where  on  the  spectrum  of  conflict  the  analysis  is 
located,  lies  in  making  a  properly  quantified  linkage 
between  C2  Measures  of  Performance  (MoP),  such 
as  communication  system  delays,  C2  Measures  of 
Measures  of  C2  Effectiveness  (MoCE),  such  as 
planning  time,  and  their  resultant  impact  on  higher  level 
Measures  of  Force  Effectiveness  (MoFE)  or  Measures 
of  Policy  Effectiveness  (MoPE),  which  capture  the 
effects  on  battle  or  operations  outcome.  These  higher 
level  MoFE/MoPE  are  required  in  order  to  be  able  to 
trade  off  investment  in  C2  systems  against  investment 
in  combat  systems  such  as  tanks  or  aircraft.  At  present, 
there  is  no  routine  way  of  making  this  linkage,  nor  any 
one  tool  that  can  be  applied  to  generate  the  required 
measures.  Flence,  all  analyses  of  C2  issues  demand 
a  high  level  of  creative  problem  structuring  and 
approach,  and  selection  and  application  of  a  range  of 
available  analysis  tools,  to  overcome  this  challenge. 

The  range  of  issues  is  very  broad  and  challenging. 
Particularly  challenging  are  the  following  issues  that 
will  be  addressed  in  more  detail  later  in  this  chapter: 

•  Representation  of  human  behaviour  (e.g.  rule- 
based,  algorithmic,  or  “human-in-the-  loop”); 

•  Homogeneous  models  versus  hierarchies/ 
federations; 
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•  Stochastic  versus  deterministic  models; 

•  Adversarial  representation; 

•  Verification,  Validation,  and  Accreditation 
(VV&A);  and 

•  The  conduct  of  sensitivity  analysis  and  other 
ways  of  dealing  with  uncertainty. 

Of  particular  importance  for  analysis  of  OOTW: 

•  Selecting  an  orchestrated  set  of  tools  that 
generate  the  required  MoM; 

•  Scoping  the  analysis  considering  tool  availability; 

•  Considering  the  human  dimension  in  tool 
selection  early  in  the  process;  and 

•  Ensuring  tools  selected  have  the  trust  and 
confidence  of  the  decisionmaker(s). 

Representation  of  Human  Behaviour: 
Rule-Based ,  Algorithmic,  or  “Human-in- 
the  Loop” 

In  developing  methods  and  tools  that  represent  the 
process  and  performance  of  C2  explicitly,  most 
approaches  until  very  recently  have  been  founded  on 
the  artificial  intelligence  (Al)  methods  of  expert 
systems.  These  represent  the  commander’s 
decisionmaking  process  (at  any  given  level  of 
command)  by  a  set  of  interacting  decision  rules.  The 
advantage  of  such  an  approach  is  that  it  is  based  on 
sound  Al  principles.  However,  in  practice  it  leads  to 
tools  which  are  large,  complex,  and  slow.  The  decision 
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rules  themselves  are,  in  many  cases,  very  scenario 
dependent  and,  as  noted  in  Chapter  6  -  Human 
Decisionmaking,  human  factors  and  organisational 
expertise  may  be  needed  on  a  project  team  to  treat 
these  issues  correctly. 

These  factors  were  not  a  problem  when  the  Cold  War 
prevailed.  There  was  sufficient  time  to  complete 
extended  analyses,  and  one  key  scenario  dominated. 
However,  in  the  post-Cold  War  environment,  such 
certainties  have  evaporated.  Indeed,  uncertainty  is  now 
one  of  the  key  drivers  of  analysis.  There  is  an 
increasing  requirement  to  consider  large  numbers  of 
scenarios  and  to  perform  a  wide  range  of  sensitivity 
analyses.  This  has  led  to  a  requirement  for  ‘lightweight,’ 
fast  running  tools,  that  can  easily  explore  a  wide  range 
of  scenarios,  yet  still  appropriately  represent  C2.  Some 
have  begun  to  explore  advanced  algorithmic  tools 
based  on  Bayesian  mathematics,  catastrophe  theory, 
and  complexity  theory.  Such  approaches  to  the 
representation  of  C2  are  at  the  core  of  a  new 
generation  of  closed  form  constructive  simulation 
models  that  are  beginning  to  be  used  for  analysis. 
(Moffat,  2002;  Moffat,  2000). 

Many  analyses  employ  “human-in-the-loop”  techniques 
in  order  to  ensure  realistic  human  performance  or  to 
check  assumptions  and  parameters.  However,  “human- 
in-the-loop”  techniques  are  expensive  and  require  the 
inclusion  of  soft  factors  and  their  attendant  MoM.  The 
introduction  of  “human-in-the  loop”  introduces  a  source 
of  variances  and  uncertainty.  The  increased  cost, 
complexity,  and  uncertainty  of  “human-in-the-loop” 
methods  often  requires  analysts  to  limit  the  use  of  these 
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techniques  rather  than  employ  them  as  the  primary 
analytical  method. 

Homogeneous  Model-Tools  versus 
Hierarchies/Federations 

In  order  to  build  an  “audit  trail”  that  traces  the 
interrelationships  among  individual  C2  systems, 
processes,  and  organisations,  as  well  as  their  impacts 
on  mission  operational  outcomes,  there  is  a  need  to 
represent  the  key  detailed  processes  involved,  such  as 
the  transmission  of  communications  across  the 
battlefield  and  the  impact  of  logistics  on  decisionmaking. 
Taking  this  as  an  example,  the  question  then  arises  as 
to  whether  all  the  transmission  media  (radio,  satellites, 
etc.),  with  their  capacities,  security  level, 
communications  protocols,  etc.,  should  be  represented 
explicitly,  or  whether  these  details  should  be  split  out  in 
a  supporting  model.  Similarly,  the  details  of  logistics 
could  be  undertaken  as  part  of  the  main  model  or  in  a 
specialised  supporting  model.  Supporting  models  could 
be  run  off-line,  providing  sets  of  input  data  to  the  main 
model  (giving  rise  to  a  model  hierarchy)  or  they  could 
be  run  in  real  time  interaction  with  the  main  model  (giving 
rise  to  a  model  federation).  In  the  off-line  mode,  the 
main  model  would  generate  demands  on  the 
communications  and  logistics  systems.  The  supporting 
models  would  check  if  these  demands  could  be 
satisfied.  If  not,  communication  delays  and  logistics 
constraints  in  the  main  model  would  be  increased,  and 
the  main  model  re-run.  This  would  have  to  be  done  a 
number  of  times  to  bring  the  main  and  supporting 
models  into  balance.  However,  such  an  approach  can 
generate  valuable  analytical  insights. 
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Figure  8.1  shows  the  main  model-tool  producing  (in 
addition  to  its  MoFE)  a  detailed  set  of  dynamic 
demands  on  the  communications  (such  as  capacity 
required  of  different  communications  systems  as  a 
function  of  simulated  time),  and  logistics  processes 
(demands  for  transport  and  key  consumables),  in  order 
to  achieve  the  assessed  levels  of  MoFE.  These  are 
then  fed  back  into  detailed  model-tools  of  the 
communications  and  logistics  infrastructure.  Those 
supporting  model-tools  can  then  be  matched  against 
the  dynamic  demand  placed  on  the  communications 
and  logistics  infrastructure  to  the  available  capacity.  If 
there  is  a  mismatch,  the  assumptions  in  the  main 
model-tools  are  adjusted  iteratively  to  bring  the  two 
model-tools  into  balance.  This  approach  is  more 
flexible  and  reactive  for  a  large  set  of  C2  assessments. 
However,  this  approach  increases  the  complexity  of 
the  architecture  (number  of  linked  sub-processes, 
failure  of  the  sub-model,  etc.) 
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Figure  8.1 .  Model  Linkage 
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A  similar  approach  can  be  applied  to  concepts  of 
operation.  In  some  models,  it  is  possible  to  describe  a 
concept  of  operations  as  a  sequence  of  standard 
missions  (e.g.  attack,  defend,  move).  These  missions 
can  then  be  analysed  to  determine  the  demands  they 
place  on  the  supporting  infrastructures.  This  can  be 
tested  off-line  to  see  if  the  infrastructure  can  cope. 
Again,  this  would  have  to  be  iterated  a  number  of  times, 
but  provides  an  ability  to  relate,  in  an  understandable 
way,  the  infrastructure  capacity  to  its  ability  to  support 
a  defined  concept  of  operations  (and  hence  battle 
outcome).  In  addition  to  the  use  of  such  hierarchies  of 
supporting  models  in  an  off-line  mode,  it  is  possible  to 
create  real-time  federations  of  such  models  to 
represent,  inter  alia,  combined  or  joint  operations. 

Stochastic  versus  Deterministic  Models 

Stochastic  and  deterministic  models  differ  in  how  they 
treat  variables.  Stochastic  models  incorporate  the 
attributes  of  the  probability  density  (or  distribution) 
functions  associated  with  model  variable  into  run  time 
calculations.  The  actual  values  of  model  variables  will 
therefore  differ  each  time  the  model  is  run.  Thus,  the 
output  of  a  stochastic  model  will  yield  different  results 
each  time  even  when  a  set  of  inputs  is  “fixed.” 
Deterministic  models  utilise  point  estimates  for  the 
values  of  model  variables,  and  hence  for  any  given 
set  of  inputs  the  model  will  produce  a  single  output. 

Stochastic  and  deterministic  models  each  have  their 
advantages.  The  selection  of  which  to  use  or  how  to 
employ  each  type  of  model  in  an  assessment  depends 
upon  the  nature  of  the  problem  and  other  elements  of 
the  solution  approach.  Whether  the  model  is  time  step 
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driven  or  event  sequence  driven  will  also  impact  on 
the  selection  of  the  appropriate  models(s). 

Deterministic  Models 

The  merits  of  a  deterministic  approach  are  that  run-times 
are  reduced,  and  there  is  a  single  ‘thread’  connecting 
the  input  data  and  the  results,  making  the  analysis  of  the 
tool  output  potentially  easier.  The  justification  for  using 
deterministic  (expected  value)  tools  is  usually  based  on 
the  assumption  that,  due  to  the  large  number  of  stochastic 
processes  involved  in  combat  on  higher  levels,  their 
results  converge  rather  quickly  to  the  mean  values 
obtained  from  stochastic  models. 

Stochastic  Models 

Stochastic  models  do  provide  significantly  more 
information  than  deterministic  tools.  They  are  deemed 
to  be  indispensable  for  an  assessment  of  the 
robustness  of  the  results  as  well  as  providing  the  “data” 
needed  for  risk  analysis.  These  additional  benefits 
come  at  the  cost  of  significantly  higher  run  time 
requirements  and  a  more  complex  analysis  task. 

Chaos  theory  shows  that  structural  variance  (or 
‘deterministic  chaos’)  can  occur  when  sets  of  decision 
rules  interact  in  the  simulation  of  a  dynamic  process. 
Small  changes  in  initial  conditions  can  lead  to  very 
different  trajectories  of  system  evolution.  Any 
simulation  model  of  combat,  with  a  representation  of 
C2,  has  to  face  this  kind  of  problem.  The  merits  of  a 
deterministic  approach  are  that  run  times  are  reduced 
and  there  is  a  single  ‘thread’  connecting  the  input  data 
and  the  results,  making  analysis  of  the  model  output 
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potentially  easier.  However,  the  representation  of  the 
C2  process  (whether  using  decision  rules  or  not)  gives 
rise  to  a  number  of  alternative  decision  options  at  any 
given  moment,  and  can  thus  potentially  give  rise  to 
such  ‘deterministic  chaos’.  If  such  effects  are  likely  to 
arise,  one  solution  is  to  use  stochastic  modelling.  The 
use  of  stochastic  sampling  in  the  model,  together  with 
multiple  replications  of  the  model,  gives  rise  to  a 
distribution  of  outcomes,  which  is  much  more  resistant 
to  such  chaotic  effects. 

US  Army  Training  And  Doctrine  Command  (TRADOC) 
has  been  experimenting  with  Deterministic  Combat 
Models.  “A  potential  alternative  solution,  when  the  issue 
under  study  warrants,  is  to  conduct  analysis  of  multiple 
runs  of  a  deterministic  model  where  the  initial  states  of 
information  systems  are  varied.”  (Bailey,  2001 ). 

Representing  Adversary  Forces 

Historically,  adversary  capabilities  and  behaviours 
were  often  fully  scripted  or  heavily  constrained.  This 
was  more  appropriate  in  Cold  War  contexts  than  it  is 
today.  However,  it  was  never  ideal  for  C2  analysis 
because  the  dynamic  interaction  among  friendly, 
adversary,  and  other  forces  is  a  critical  element  of  C2 
representation.  Today,  much  more  robust  adversary 
representation  of  operational  capabilities  and  choices 
are  employed  and  indeed  are  necessary.  Analysts 
must  consider  not  only  a  range  of  scenarios,  but  also 
the  range  of  possible  adversary  actions  and  reactions. 
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Verification ,  Validation ,  and 
Accreditation  (VV&A) 

VV&A  has  historically  been  a  challenge  for  model 
development  efforts,  but  is  particularly  challenging  for 
C2  modelling.  This  is  due  to  the  variability  inherent  in 
most  C2  processes,  especially  those  that  involve  the 
human  aspects  of  information  processing  and 
decisionmaking.  The  approach  to  VV&A  needs  to  be 
carefully  considered,  particularly  in  light  of  the  need  to 
assess  future  C2  systems  and  capabilities  in  association 
with  new  concepts  of  operation,  new  organisational 
forms,  new  doctrine  and  asymmetrical  adversaries. 

Selecting  an  Orchestrated  Set  of  Tools 

The  natural  tendency  of  an  analyst  is  to  simplify  a 
problem.  Part  of  that  simplification  is  to  select  a  tool, 
preferably  only  one,  which  will  meet  the  analysis 
requirements.  In  the  analysis  of  C2  of  combat 
operations,  this  may  be  possible  if  the  analysis  is 
properly  scoped.  In  the  analysis  of  OOTW  C2,  the 
issues  are  typically  too  numerous,  the  variables  too 
confounding  and  the  scope  too  broad  for  one  tool  to 
satisfy  all  analysis  requirements.  An  orchestrated  set 
of  complementary  tools  will  normally  be  required. 

Scoping  the  Analysis  Considering 
Tool  Availability 

An  analyst  must  always  scope  the  analysis  during 
problem  formulation  to  enable  it  to  be  accomplished 
within  available  resource  constraints.  During  problem 
formulation,  however,  the  consideration  of  available 
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tools  has  typically  not  been  a  driving  factor.  Also,  in 
the  analysis  of  C2  combat  operations  this  may  not  be 
a  problem  because  selection  of  the  tool(s)  to  be  used 
is  more  obvious  based  on  past  experience.  However, 
for  OOTW  C2  analysis,  the  availability  of  tools,  and 
their  orchestration,  requires  more  consideration  early 
in  the  process  (i.e.  during  problem  formulation). 

Consideration  of  the  Human  Dimension 

C2,  by  its  very  nature,  is  closely  linked  to  human 
behaviour,  and  its  analysis  requires  careful  consideration 
and  inclusion  of  the  human  dimension.  Often,  as  a  way 
of  simplifying  the  analysis,  the  C2  assessment  team 
eliminates  these  considerations  by  assuming  that  human 
commanders  and  their  staffs  are  not  affected  by  their 
environment  and  will  always  make  the  best  decision,  and 
the  same  decision,  given  the  required  information.  This 
is  unsafe  and  not  good  practice.  This  has  been  especially 
true  for  the  analysis  of  OOTW  C2,  even  though  this 
analysis  is  often  more  impacted  by  the  human  dimension 
than  C2  for  combat  operations. 

Ensuring  Trust  and  Confidence  in  the  Tools 

Analysts  select  and  apply  the  tools  of  their  trade  based 
on  what  those  tools  can  do  for  them  in  accomplishing 
their  analysis  objectives.  Over  time  they  become 
comfortable  with  certain  tools  and  the  customers  for 
their  analysis  also  develop  a  trust  in  the  tools  and 
confidence  that  they  will  produce  valid  results  for  them. 
For  OOTW  C2  analysis,  given  that  the  tools  are  more 
numerous,  must  be  orchestrated  to  work  together,  and 
are  sometimes  unknown  or  not  understood  by  the 
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customer,  the  development  of  trust  and  confidence  in 
these  tools  is  difficult  to  achieve. 

New  Methods  and  Emerging  Practices 

Given  the  inherent  problems  associated  with  the  issues 
described  previously,  a  best  practice  for  the  application 
of  analysis  tools  for  C2  analysis  is  still  emerging.  This 
best  practice  is  described  below,  first  for  tools  in 
general,  then  for  models,  both  at  the  model  level  itself 
and  at  the  algorithm  level. 

Selection  of  Methods  and  Tools 

The  selection  of  tools  to  apply  to  C2  analysis  should 
be  based  both  on  evaluation  of  the  candidate  tools 
themselves  against  a  set  of  evaluation  criteria  and  on 
consideration  of  the  type  of  study  to  be  undertaken. 
The  evaluation  criteria  for  tool  selection  includes  criteria 
related  to  the  functionality  of  the  candidate  tool  and  to 
the  performance  of  the  candidate  tool.  The  following 
are  established  evaluation  criteria  for  tool  selection. 
They  are  as  applicable  to  tool  selection  for  OOTW  C2 
as  for  combat  C2  analysis. 

•  Functionality-related  tool  selection  criteria: 

•  Resolution:  the  level  of  detail  in 
representation  of  entities  within  the  tool; 

•  Completeness/scope:  the  extent  to  which  the 
tool  is  able  to  address  analysis  issues; 

•  Functionality:  the  extent  to  which  the  tool 
represents  the  full  range  of  functions; 
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■  Explicitness:  the  ability  of  the  tool  to  explicitly 
represent  required  entities; 

•  MoM  Generation:  the  ability  of  the  tool  to 
generate  the  MoM  required; 

•  VV&A:  the  determination  of  whether  the  tool 
has  been  verified,  validated,  and/or 
accredited  for  it  intended  use;  (Note:  Not  all 
NATO  member  nations  recognise  the  term 
“Accreditation.”  Accreditation  here  refers  to 
some  form  of  formal  approval  to  use  the 
model  for  the  analysis  intended.) 

•  Performance-related  tool  selection  criteria: 

•  Responsiveness:  the  amount  of  time  between 
request  and  receipt  of  information; 

•  Simplicity:  the  ease  of  preparation  and  use 
the  tools; 

•  Preparation/use  time:  the  length  of  time 
necessary  to  prepare  and  use  the  tool; 

•  Data  availability  and  parameters:  the  ease  in 
acquiring  or  generating  the  necessary  data  or 
parameters  for  tool  use; 

•  Interoperability:  the  ability  of  the  tool  to 
interoperate  with  other  tools; 

•  Resource  requirements:  the  amount  of 
resources  (time,  personnel,  and  funds) 
required;  and 

•  Credibility:  the  extent  to  which  the  customers 
and  users  accept  tool  results. 
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Using  Models 

The  potential  for  exploiting  recent  advances  in 
mathematics  in  order  to  create  fast  running  model- 
tools  was  noted  earlier.  Such  models  have  exploited 
emerging  approaches,  such  as  complexity  theory, 
chaos  theory,  catastrophe  theory,  and  game  theory  in 
order  to  produce  a  ‘good  enough’  representation.  They 
can  be  used  to  complement  more  complex,  detailed 
models  of  the  problem  area.  In  many  cases,  a  tailoring 
of  models,  or  other  tools,  will  be  required  to  properly 
address  the  analysis  issues  at  hand.  (Moffat,  2002; 
Moffat,  2000). 

Model  Federations 

A  number  of  new  approaches  share  a  key  set  of 
characteristics.  First,  an  object-oriented  approach 
within  the  model-tool  allows  different  objects  to  be 
brought  together  to  represent  the  complete  command 
process,  rather  like  ‘Lego™  bricks.’  Such  a  philosophy 
also  encourages  the  development  of  model-tools 
based  on  holistic  and  evolutionary  principles.  In  other 
words,  always  capture  a  complete  model  of  the 
process,  including  the  parts  whose  representation  is 
still  unclear.  As  understanding  develops,  improve  those 
parts  (or  objects)  which  were  rudimentary  at  the  start. 
At  the  next  level  up,  the  use  of  run-time  interfacing 
allows  different  model-tools  to  be  brought  together  to 
create  a  federation  to  represent  the  process  under 
study.  This  federation  then  may  also  have  to  be 
integrated  with  the  use  of  a  mix  of  tools,  which  include 
techniques  other  than  modelling,  to  fully  address  the 
study  issues. 
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Agent-Oriented  Modelling 

A  second  key  aspect  is  the  description  and 
representation  of  the  C2  process  through  agent 
modelling  and  programming  techniques.  Modelling  of 
the  C2  process  as  a  group  of  agents,  based  on  artificial 
intelligence  concepts,  favours  the  capture  of  the 
cognitive  nature  of  command  tasks.  Agents  can  be 
implemented,  in  an  object-oriented  environment,  as 
either  objects  (e.g.,  actor  or  “applet”  type  of  agents), 
or  aggregates  of  objects  (coarse-grain  agents).  Such 
agents  interact  with  each  other  through  a  messaging 
infrastructure.  The  term  “agent-oriented  modelling”  is 
suggested  as  a  way  of  capturing  this  idea. 

Linking  of  Performance  Model-Tools  to 
Effectiveness  Tools 

This  third  idea,  used  in  a  number  of  NATO  countries, 
uses  a  structured  hierarchy  of  model-tools  to  create 
an  audit  trail  from  C2  systems,  processes,  and 
organisations  through  to  battle  operations  outcome. 
The  idea  is  to  create  supporting  performance  level 
model-tools  of  particular  aspects  of  the  process  (e.g., 
communications,  logistics)  which  can  be  examined  at 
the  performance  level.  These  then  form  inputs  to  higher 
level  force  on  force  models.  This  ensures  that  the 
combat  models  themselves  do  not  become  overly 
complex.  We  have  already  discussed  this  in  the  sense 
of  hierarchies  of  support  models. 

For  example,  in  Figure  8.2,  a  detailed  model  of  the 
intelligence  system  can  be  very  complex,  if  we  wish  to 
take  account  flow  of  intelligence  requirements,  tasking, 
collection  processes,  fusion  processes,  and 
intelligence  products.  In  order  to  analyse  the  impact 
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of  intelligence,  it  is  important  to  have  all  of  this  detail, 
but  it  does  not  necessarily  have  to  be  represented 
explicitly  in  the  main  model.  A  supporting  model-tool 
which  captures  all  of  this  detail  can  be  created  (or  used 
if  one  already  exists)  in  order  to  produce  outputs  at 
the  MoCE  level,  such  as  speed  and  quality  of 
intelligence.  These  can  then  form  inputs  to  the  main 
simulation  model.  The  main  model-tool  then  takes 
these  into  account  in  producing  its  own  outputs.  These 
will  now  be  at  the  MoFE  level.  Examples  are  friendly 
casualty  levels,  adversary  attrition,  and  time  to  achieve 
military  objectives. 


Scanning  Scenario  Space 

The  use  of  very  fast  model-tools  to  scan  the  overall 
space  of  possibilities  and  to  identify  areas  of  concern 
for  further  analysis  appears  to  give  a  good  balance 
between  the  use  of  simple  and  complex  modelling 
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approaches.  Fast  models,  which  are  simpler  but  may 
have  less  analytic  depth,  allow  the  analyst  to  scope 
the  problem  and  determine  the  degree  of  complexity 
a  model-tool  must  represent  in  order  to  conduct  the 
desired  level  of  analysis. 

Decisionmaking  Process 

It  is  important  to  have  a  proper  representation  of  the 
decisionmaking  process  in  order  to  establish  the  link 
from  C2  performance  through  MoE  to  overall  MoFE 
and  to  represent  information  operations  (10)  effects 
such  as  Counter  C2  or  digitisation  of  the  battlespace. 
Representation  of  the  decisionmaking  process  itself, 
however,  remains  difficult  because  of  the  difficulty  in 
representing  human  performance,  command  styles, 
and  organisational  relationships. 

Parameter  Development  Context 

Finally,  it  may  become  necessary  to  generate  new  or 
additional  data  to  validate  new  or  existing  model-tools 
to  incorporate  C2  factors.  This  may  be  especially  true 
in  the  case  of  integrating  soft  factors  into  C2  analysis. 
Possible  methods  include  field  trials,  “model-test- 
model”,  or  advanced  warfighting  experiments.  Field 
trials  are  used  if  uncertainty  revolves  around 
measurable  factors  that  are  only  observable  in  the  field 
or  are  not  reproducible  in  the  laboratory.  Model-Test- 
Model  (M-T-M)  or  Model-Exercise-Model  (M-E-M)  is 
used  as  part  of  an  iterative  process  to  develop  and 
apply  systematically  more  in-depth  and  sophisticated 
model-tools  and,  in  some  cases,  more  simplistic 
model-tools  to  increase  their  validity.  The  original 
model-tool  is  executed,  modified  based  on  results  of 
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the  test  or  experiment,  and  executed  again  until  it  has 
developed  a  sufficient  representation  of  a  complex 
process.  Experiments,  like  advanced  warfighting 
experiments,  are  useful  in  modelling  new,  large  scale, 
and  complex  interactions  for  which  little  data  or  few 
validated  tools  exist.  Each  approach  requires  additional 
time  and  resources,  and  the  data  sets  may  not  be 
validated  for  some  time. 

C2  Modelling  Guidelines 

A  number  of  common  ideas  have  emerged  which  are 
worth  consideration  for  new  modelling  and  tool 
developments.  Figure  8.3  shows  how  each  of  the 
following  common  ideas  are  represented  within  an 
ideal  command  and  control  model: 

•  Understanding  of  adversary  intent  can  be 
represented  by  having  a  number  of  prescribed 
intents  or  options,  which  are  updated  in  an 
advanced  data  architecture,  or  Bayesian  way,  as 
more  information  becomes  available; 

•  Representing  headquarters  explicitly  in  the  model- 
tool  allows  proper  representation  of  Information 
Warfare  (IW)  effects  such  as  counter  C2; 

•  Explicit  representation  of  the  “recognised  picture” 
within  each  headquarters  (HQ)  allows  the  model- 
tool  to  run  based  on  different  perceptions  by  each 
individual  unit  on  each  side  represented.  This 
allows  the  effects  of  aspects  such  as  deception, 
shock,  and  surprise  to  be  explicitly  considered; 

•  Represent  information  as  a  commodity.  This 
consideration  is  the  most  critical  and  difficult  to 
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implement,  but  is  the  foundation  for  the  other 
guidelines,  as  well  as  for  the  model  itself. 
Information  should  be  considered  as  a  resource 
that  can  be  collected,  processed,  and 
disseminated.  It  includes  information  about 
adversary,  friendly,  and  other  forces, 
considerations  such  as  political-military  factors  and 
rules  of  engagement  (ROE),  as  well  as 
environmental  information  such  as  weather  and 
terrain.  Information  should  posses  dynamic  values 
such  as  accuracy,  relevance,  timeliness, 
completeness,  and  precision.  These  values  should 
in  some  way  affect  other  activities  within  the  model, 
to  include,  when  appropriate,  combat  functions; 
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Figure  8.3.  Modelling  Guidelines 


•  Represent  the  realistic  flow  of  information 
throughout  the  operational  environment. 
Information  has  a  specific  source,  and  that  source 
is  usually  not  the  end  user  of  the  information.  A 
requirement  exists,  therefore,  to  move  information 
from  one  place  to  another  in  the  operational 
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environment.  Communications  systems  of  all 
forms  exist  to  accomplish  this  movement.  These 
systems  can  be  analogue  or  digital.  Information 
can  be  lost  and/or  degraded  as  it  flows  around  the 
operational  environment.  The  model-tool  should 
represent  the  communications  systems  and 
account  for  these  degradation  factors  as  it 
represents  information  flow; 

•  Represent  the  collection  of  information  from 
multiple  sources  and  tasking  of  information 
collection  assets.  This  guideline  applies  equally 
to  adversary,  neutral,  and  friendly  and  other 
force  information.  For  the  collection  of  adversary 
and  other  force  information,  the  model-tool 
should  represent  a  full  suite  of  sensors  and 
information  collection  systems,  and  the  ability  of 
these  systems  to  be  tasked  to  collect  specific 
information.  For  the  collection  of  friendly 
information,  this  consideration  is  just  as  critical. 
Knowledge  of  one’s  own  capability  in  combat,  as 
well  as  that  of  the  adversary  and  other  forces,  is 
essential  for  effective  decisionmaking; 

•  Represent  the  processing  of  information. 
Information  is  rarely  valuable  in  its  original  form.  It 
usually  has  to  be  processed  in  some  way.  Typical 
processing  requirements  include  filtering, 
correlation,  aggregation,  disaggregation,  and 
fusion  of  information.  These  processes  can  be 
accomplished  by  either  manual  or  automated 
means.  The  ability,  or  inability,  to  properly  process 
information,  and  the  time  it  takes  can  have  a 
direct  bearing  on  combat  operational  outcome; 
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•  Represent  C2  systems  as  entities  in  the 
operational  environment.  C2  systems  perform 
information  collection,  processing,  dissemination, 
and  display  functions.  They  should  be  explicitly 
represented  as  entities  that  can  be  targeted, 
degraded,  and/or  destroyed  by  either  physical  or 
non-physical  means.  Additionally,  the  model-tool 
should  account  for  continuity  of  operations  of 
critical  functions  during  periods  of  system  failure 
or  degradation; 

•  Represent  unit  perceptions  built,  updated,  and 
validated  from  the  information  available  to  the 
unit  from  its  information  systems.  This  is  a 
critical  requirement.  Each  unit  should  have  its 
own  perceptions,  gaining  knowledge  from 
superior,  subordinate,  or  adjacent  units  only 
when  appropriate; 

•  Represent  commander’s  decisions  based  on  the 
unit’s  perception  of  the  battlefield.  Each  unit 
should  act  based  on  what  it  perceives  the 
situation  to  be  informed  by  its  commander’s 
intent  of  mission,  goals,  constraints,  and  biases, 
not  based  on  ground  truth  available  within  the 
model.  When  a  unit  takes  action  based  on 
inaccurate  perceptions,  it  should  suffer  the 
appropriate  consequences;  and 

•  Represent  10  for  each/all  combatants.  With 
information  so  critical  to  combat  operations 
outcome,  the  model-tool  should  be  able  to 
represent  the  deliberate  attack  and  protection  of 
information,  information  systems,  and  decisions. 
This  applies  to  all  sides  represented  in  the  model. 
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As  shown  in  Figure  8.3,  explicit  representation  of  these 
information  operations  elements  within  a  command 
and  control  will  facilitate  the  assessment  of  human 
factors,  psychological  operations,  deception,  C2 
systems  effectiveness,  and  staff  structure  issues. 

Conclusions 

The  following  conclusions  are  made  regarding  the 
strengths  and  weaknesses  of  current  C2  tools/ 
modelling  approaches. 

Strengths  in  Current  02  Tools/Modelling 

An  assessment  of  current  C2  modelling  approaches 
employed  against  the  guidelines  above  show  that, 
while  they  may  not  yet  be  satisfied,  there  are  some 
strengths  in  the  C2  tools/modelling  approaches 
currently  being  implemented: 

•  There  is  a  common  understanding  of  issues.  This 
has  not  always  been  the  case.  With  the  inherent 
complexity  of  C2,  as  well  as  the  challenges  in 
modelling  such  a  complex  subject,  there  has 
been  a  tendency  in  the  past  to  ignore  the 
subject,  or  just  to  accept  it  as  something  that  is 
too  complex  to  address.  This  does  not  seem  to 
be  the  case  now.  Perhaps  through  the 
emergence  of  new  technologies  and  through  the 
work  of  groups  such  as  the  NATO  SAS  groups, 
the  analysis  and  modelling  of  C2  is  now 
considered  possible.  Member  nations  now  seem 
to  have  both  an  understanding  of  C2  and  its 
importance  to  combat  and  OOTW  operations 
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and  a  common  understanding  of  the  modelling 
challenges  that  exist. 

•  There  is  wide  application  of  C2  modelling. 
Member  nations  now  apply  C2  modelling  and 
analysis  to  a  wide  range  of  issues.  These  issues 
include  those  associated  with  investment, 
requirement  identification,  force  structuring  and 
operational  support.  In  all  of  these  areas,  there  is 
understanding  of  the  sensitivity  and  criticality  of 
C2  to  the  proper  analysis  of  combat  operations 
outcome.  Selection  of  the  model-tools  to  apply  to 
a  particular  problem  should  be  based  on 
evaluation  of  specific  criteria,  as  discussed 
previously  in  this  chapter. 

•  Although  each  nation  develops  its  models  for 
different  purposes  and  tailors  their  models  and  other 
tools  for  specific  issues,  there  exists  a  commonality 
of  approaches  in  different  nations  that  serves  to 
strengthen  their  collective  merits.  These  common 
approaches  are  an  outgrowth  of  the  modelling 
technologies  now  available,  but  also  result  from 
shared  experiences  by  member  nations. 

•  Most  of  the  progress  and  success  in  C2 
modelling  has  been  with  regard  to  high-intensity 
combat.  This  is  perhaps  due  to  the  belief  by 
many  that  a  high-intensity  combat  scenario  is  still 
most  appropriate  for  the  analysis  of  combat, 
particularly  for  analysis  of  primary  combat 
systems.  Progress,  therefore,  has  been  focused 
on  embellishing  high-intensity  combat  analysis 
with  C2  improvements  to  models.  Unfortunately, 
low-intensity  combat  and  OOTW  modelling  and 
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analysis  have  not  received  the  same  level  of 
attention  until  recently. 

•  There  is  wide  use  of  evolutionary  development 
approaches.  After  many  years  of  neglect,  a 
problem  as  complex  and  difficult  as  C2  modelling 
requires  years  of  focused  research  and 
development.  There  are  no  simple  fixes  to  the 
problem.  It  is  evident  that  member  nations 
recognise  this  and  are  willing  to  approach  the 
problem  in  an  incremental  manner,  applying 
evolutionary  approaches. 

•  There  is  progress  in  linking  and  federating 
models.  Significant  progress  has  been  made  by 
several  nations  in  linking  performance  model- 
tools  with  combat  effectiveness  models,  either 
directly  or  through  off-line  approaches. 
Additionally,  creating  federations  of  models 
through  standard  interface  protocols  has 
significantly  improved  the  use,  and  reuse,  of 
existing  models  and  has  provided  a  promising 
approach  for  future  modelling.  The  inherent 
difficulties  in  federating  models,  given  today’s 
state  of  the  art,  must  be  considered  when 
contemplating  this  approach. 

•  There  is  progress  in  modelling  “soft  factors.” 
Several  nations  have  made  real  progress  in 
modelling  phenomena  that  have  non-physical,  or 
soft  impact,  on  combat  operations  outcome. 
Among  these  factors  are  morale,  fatigue,  and 
training  proficiency.  These  and  other  soft  factors 
have  increased  importance  on  combat 
operations  outcome  as  C2  modelling  improves 
combat  models. 
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•  Standard  interface  protocols,  data  standards,  and 
other  standards  either  now  exist,  or  are  under 
development.  These  standards  serve  to  make  this 
difficult  task  easier.  Continued  development  of  such 
standards  is  envisioned  for  the  future  is  essential. 

•  There  is  widespread  use  of  Commercial  Off  The 
Shelf  (COTS)  products.  These  products  are 
generally  available  to  all  member  nations.  This 
use  of  COTS  has  served  to  help  further 
standardise  individual  modelling  approaches  and 
will  continue  to  do  so  in  the  future. 

•  There  is  application  of  new  information 
technologies.  New  technologies,  such  as  those 
supporting  animation,  have  been  applied  to  the 
challenge  of  C2  modelling  simulation,  and 
analysis.  Additional  technological  advancements 
will  no  doubt  continue  and  will  be  similarly 
applied  to  this  problem. 

Weaknesses  in  Current  C2  Tools/Modelling 

An  assessment  of  current  C2  modelling  approaches 
employed  against  the  guidelines  above  show  that 
weaknesses  exist  in  current  approaches.  These 
weaknesses,  rather  than  being  enumerated  here,  are 
expressed  as  challenges  below. 

Recommendations 

It  is  recommended  that  analysts  take  advantage  of  the 
strengths  available  in  current  approaches  and  in  the 
new  methods  that  are  evolving.  They  should  also  be 
aware  of  the  challenges  that  must  still  be  resolved  and 
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should  attempt  to  play  their  part  in  helping  address  those 
challenges  both  through  study  activities  and  research. 

Challenges:  OOTW 

As  discussed  previously,  recent  world  events  and 
current  projections  call  for  an  emphasis  on  OOTW. 
C2  in  these  environments  can  be  quite  different  and 
may  require  fresh  C2  tool/modelling  approaches  to  link 
C2  to  outcome  in  these  environments.  The  following 
challenges  with  tools-models  for  C2  Assessment  exist: 

•  Orchestrating  a  set  of  applicable  tools.  Because 
there  is  no  one  universally  accepted  tool  that  will 
satisfy  OOTW  C2  analysis  requirements,  a  set  of 
tools  must  be  selected,  based  on  evaluation  of 
potential  tools  against  selection  criteria,  and 
applied  to  the  analysis.  Orchestrating  a  set  of 
tools  that  complement  the  strengths  and 
weaknesses  of  each  to  satisfy  the  analysis 
requirements  is  difficult.  Proper  consideration  of 
assumptions  and  constraints  during  tool 
selection,  and  careful  scoping  the  analysis 
issues,  will  help  to  simplify  the  orchestration  of 
tools.  Additionally,  it  must  be  remembered  that 
not  only  the  tools,  but  trained,  skilled  users  of  the 
tools  are  required. 

•  Breadth  of  tool  application.  Because  of  the 
complexity  of  OOTW  analysis,  the  full  set  of 
potential  tools  should  be  considered  for 
application  throughout  the  study  process,  from 
problem  formulation  through  sensitivity  analysis. 
Additionally,  the  more  subjective  nature  of 
OOTW  dictates  that  tools,  heretofore  not 
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typically  applied  to  the  analysis  of  C2,  be 
considered  for  use  throughout  the  study  process. 

•  Relationship  of  tools  to  data  availability  and  MoM 
generation.  Tools  must  be  selected  for  OOTW 
C2  analysis  that  have  necessary  data  available, 
or  able  to  be  generated  or  obtained  from  the 
application  of  other  tools.  Reliance  on  more 
subjective  data,  especially  for  higher  C2 
echelons,  may  be  necessary.  Additionally,  tools 
must  be  selected  that  generate  the  MoMs  that 
will  help  answer  the  study  issues  at  hand.  The 
OOTW  C2  analyst  will  be  challenged  to  be 
creative  in  establishing  the  strong  relationship 
between  data,  tools,  and  MoM  required  for  a 
successful  OOTW  C2  analysis.  It  must  be 
remembered  that  analysts,  not  the  tools 
themselves,  answer  the  analysis  issues. 

•  Consideration  of  M-E-M  or  M-T-M.  Capitalisation 
on  testing  or  training  events  for  C2  analysis 
purposes  can  be  highly  beneficial,  both  from  a 
resource  and  analytic  point  of  view,  but  doing  so 
can  be  difficult.  Such  approaches  are  more 
subjective,  take  more  human  involvement,  and 
are  inherently  more  complex  than  more  classical 
analysis  tools/approaches.  The  advantages  of 
having  live  players/subjects  in  the  analysis, 
however,  generally  outweigh  the  disadvantages. 
These  approaches  are  becoming  more 
commonplace  and  more  accepted  as  a 
legitimate  analysis  approach,  especially  for  more 
subjective  issues  such  as  those  associated  with 
OOTW  C2. 
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•  Sharing  of  tools  between  different  communities. 
Given  the  nature  of  analysis  of  OOTW  C2  and  the 
emergence  of  M-E-M  and  M-T-M  approaches,  the 
boundaries  between  the  testing,  training,  analytic, 
and  operational  communities  are  blurring  and  tools 
once  considered  for  use  in  only  one  of  these 
communities  are  finding  application  in  another.  The 
sharing  of  available  tools  among  these  communities 
is  considered  even  more  appropriate  now  with  the 
rise  in  importance  of  complex  OOTW  C2  analysis 
requirements.  This  sharing,  however,  is  difficult  due 
to  differences  in  terminology,  as  well  as  cultural 
differences  between  the  different  communities. 

•  Reuse  of  operational  schemas  and  data  models. 
OOTW  typically  involves  many  more 
organisational  entities  than  combat  operations, 
making  interoperability  of  organisations  and 
systems  critical  to  successful  operations.  C2 
analysts,  therefore,  are  challenged  to  ensure 
they  use,  to  the  extent  possible,  existing 
operational  schemas,  such  as  orders  and 
reports,  and  data  models  used  for  C3I  systems 
integration,  whenever  possible  to  further 
standardization  and  interoperability  goals. 

•  Analysis  of  architectures.  Technical,  operational, 
and  system  architectures  are  sometimes 
developed  in  order  to  facilitate  integration  and 
interoperability  of  C3I  systems.  For  NATO, 
required  architecture  frameworks  are  contained 
in  the  NATO  C3  Interoperability  Management 
Plan  (NIMP),  Volume  II.  Considerable  challenges 
exist  in  developing  and  applying  analytic  tools  for 
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evaluation  architectures,  to  include  analysis  of 
alternative  architectures  and  their  implications. 

•  Management  of  customer  expectations  and 
relations.  OOTW  C2  analysis  is  so  difficult  that  it 
is  necessary  for  the  analyst  to  ensure  the 
customer  for  the  analysis  understands  the 
inherent  difficulty  and  to  attempt  to  manage  the 
customer  expectations  for  the  analysis.  This 
includes  informing  the  customer  of  the  tools  to  be 
employed  and  to  gain  an  understanding  and  trust 
in  those  tools  by  the  customer.  It  also  implies  a 
special  relationship  between  analyst  and 
customer  for  OOTW  C2  analysis.  This  is  even 
more  critical  when  the  customer  is  a  subject  of  the 
analysis,  such  as  when  a  commander  has  his/her 
own  command  analysed. 

Challenges:  Modelling  C2 

The  following  challenges  with  modelling  C2  exist: 

•  Better  representation  of  cognitive  processes.  C2 
can  be  incorporated  at  one  level  of  resolution  in 
combat  tools  through  representation  of  the 
effects  of  particular  decisions.  At  another  level, 
representation  of  the  decision  process  itself  is 
desirable.  It  would  enable  alternative 
decisionmaking  styles  and  the  effects  of  soft 
factors  such  as  stress,  training  level,  fatigue,  and 
morale  to  be  more  easily  assessed.  These 
factors  become  more  important  as  the  full  range 
of  10  representation  is  attempted; 

•  Long  standing  challenges  associated  with  both 
stochastic  and  deterministic  models.  The 
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advantages  and  disadvantages  associated  with 
stochastic  and  deterministic  modelling 
approaches  will  remain  as  C2  modelling 
improves.  The  objective  is  to  select  the  best 
modelling  approach  for  the  issue  at  hand. 
Recognising  the  inherent  advantages  and 
disadvantages,  and  then  capitalising  on  the 
advantages  while  minimising  the  disadvantages, 
are  the  challenges; 

•  Better  standard  definition  of  C2  terms.  This 
challenge  has  plagued  the  C2  community  for 
many  years,  because  of  both  the  scope  and 
complexity  of  the  subject.  This  is  particularly  true 
across  service  boundaries  and  across  the 
international  community.  A  standard  set  of 
definitions  would  greatly  simplify  the  C2 
modelling  challenge; 

•  The  definition  and  application  of  C2  scope.  This 
challenge,  related  to  the  previous  challenge,  is 
especially  critical  to  modellers.  The  C2  of  a 
fighter  aircraft  or  a  carrier  group  is  very  different 
from  the  C2  of  an  army  corps  or  an  army  squad. 
On  the  other  hand,  there  are  C2  aspects  of  each 
of  these  combat  elements  that  are  similar. 
Modelling  of  C2,  however,  can  be  vastly  different 
in  each  case.  The  scope  of  each  modelling 
undertaking  must  be  properly  considered  and 
discipline  must  be  applied  throughout  model 
development  to  focus  on  the  proper  scope.  Once 
the  scope  is  established,  a  mix  of  tools  may  be 
required  to  address  the  full  scope  of  the  analysis; 

•  Multiple  application  of  C2  model-tools  to  analysis, 
training,  and  operational  requirements.  C2 
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phenomena  are  relatively  constant  whether  they 
exist  within  the  analytic,  training,  or  operational 
environment,  and  they  should  be  consistently 
modelled  in  each  environment.  This  fact,  as  well 
as  the  obvious  need  to  conserve  expensive 
model-tool  development  resources  wherever 
possible,  leads  to  the  challenge  of  developing  C2 
models,  or  at  least  component  software  modules 
that  can  be  used  to  support  analysis,  training, 
and  operational  requirements; 

•  Level  of  resource  application  to  the  breadth  and 
depth  of  C2  modelling.  Because  of  the  large 
scope  of  C2,  there  has  been  a  tendency  by 
some  to  model  C2  at  great  breadth  (multiple 
applications),  at  the  expense  of  modelling  C2 
phenomena  at  a  corresponding  depth.  In  a 
constrained  resource  world,  sufficient  resources 
are  not  usually  available  for  both.  The  challenge 
is  to  either  apply  sufficient  resources  or  to 
recognise  the  shortfall  and  to  level  the  available 
resources  across  the  breadth  and  depth  of  the 
problem.  Models  and  other  tools  must,  therefore, 
be  tailored  to  the  extent  possible  to  fit  the  study 
issues  being  addressed; 

•  Differences  in  the  level  of  modelling  of  friendly 
and  adversary  forces.  Many  combat  models  do 
not  represent  adversary  forces  to  the  same  level 
of  resolution  as  friendly  forces.  In  the  past,  there 
may  have  been  good  reasons  for  this.  Besides 
the  obvious  resource  savings,  the  lack  of  C2 
representation  often  precluded  further 
representation  of  adversary  forces.  Valid 
representation  of  C2,  to  include  full  play  of  10 
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such  as  deception  and  psychological  operations, 
will  require  equal  representation  across  both 
adversary  and  friendly  forces,  as  well  as  any 
other  supporting  or  neutral  forces  in  the 
simulation.  All  discussion  and  recommendations 
in  the  COBP,  therefore,  are  equally  applicable  to 
modelling  of  adversary  forces  as  it  is  to 
modelling  friendly  force  C2.  This  represents  a 
significant  challenge  to  many  modelling  efforts; 

•  Continuing  lack  of  “soft  factor”  representation  and 
data.  As  discussed  previously,  a  robust  C2 
representation  in  combat  models  will  permit  soft 
factors  to  be  better  represented.  The  bigger 
challenge,  perhaps,  may  not  be  the  modelling 
methodology  itself,  but  the  acquisition  of  data  to 
support  it.  The  effects  of  such  things  as  stress, 
training  proficiency,  morale,  fatigue,  and  shock,  for 
example,  necessitate  new  data  generation 
approaches,  which  will  take  some  years  to 
implement.  The  tasks  of  VV&A  and  Verification, 
Validation,  and  Certification  (VV&C)  are  most 
severe  in  their  soft  factor  arena.  The  certification  of 
soft  factor  data,  as  well  as  most  all  C2-related  data, 
is  particularly  difficult  to  achieve.  Innovative  and 
focused  C2  data  VV&C  programs  are  required; 

•  VV&A  of  C2  model-tools  and  the  parameters  that 
drive  them.  This  is  always  a  challenge  for  model- 
tool  development  efforts,  but  is  particularly 
challenging  for  C2  modelling,  due  to  the  variability 
inherent  in  most  C2  processes,  especially  those 
that  involve  the  human  aspects  of  information 
processing  and  decisionmaking;  and 
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•  Sensitivity  Analysis.  The  challenges  associated 
with  the  proper  conduct  of  sensitivity  analysis  of 
C2  is  as  great  as,  or  perhaps  greater  than,  that 
associated  with  other  analyses.  This  is  because 
of  the  uncertainty  associated  with  C2  itself,  and 
the  relatively  immature  modelling  of  C2  that 
exists  today.  Innovative,  yet  cost-effective 
approaches  to  sensitivity  analysis  are  required. 

Chapter  8  Acronyms 

ACCESS- Army  Command  and  Control  Evaluation  System 

Al- Artificial  Intelligence 

C2  -  Command  and  Control 

CPX-  Command  Post  Exercise 

FTX-  Field  Post  Exercise 

HEAT  -  Headquarters  Effectiveness  Assessment  System 

HQ  -  Headquarters 

MAPEX-  Map  Exercise 

M-E-M  -  Model- Exercise-Model 

MoCE  -  Measures  of  C2  Effectiveness 

MoFE  -  Measures  of  Force  Effectiveness 

MoP  -  Measures  of  Performance 

MoPE  -  Measures  of  Policy  Effectiveness 

M-T-M  -  Model-Test-Model 

ROE  -  Rules  of  Engagement 
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OOTW  -  Operations  Other  Than  War 
STAFFEX-  Staff  Exercise 

TRADOC  -  US  Army  Training  And  Doctrine  Command 
W&A-  Verification,  Validation,  and  Accreditation 
W&C  -  Verification,  Validation,  &  Certification 
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Data 


Know  the  enemy  and  know  yourself;  in 
a  hundred  battles  you  will  never  be  in 
peril.  When  you  are  ignorant  of  the 
enemy  but  know  yourself,  your 
chances  of  winning  or  losing  are 
equal.  If  ignorant  both  of  your  enemy 
and  of  yourself,  you  are  certain  in 
every  battle  to  be  in  peril. 

— SunTzu,  The  Art  of  War 
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Definitions 

Data  are  factual  information  that  are  organised  for 
analysis  and  in  a  form  suitable  for  machine 
processing.  Data  are  usually  thought  of  as  anchoring 
an  epistemological  scale  with  understanding  or  wisdom 
anchoring  the  other  end.  Information  is  data  that  have 
been  put  into  context. 

Metadata  are  “information  about  information.” 
Metadata  can  describe  data  as  well  as  metadata, 
therefore  there  are  several  levels  of  metadata  possible. 
An  example  for  metadata  on  a  higher  level  is  the 
reliability  of  the  source  that  data  have  been  derived 
from.  In  general,  metadata  are  documentation  of  the 
attributes  of  data  directly  attached  to  the  data,  and 
therefore  can  be  archived  along  with  the  data. 

Role  of  Data 

The  role  and  importance  of  data  in  C2  assessment  is 
underestimated  by  many  people,  often  including  the 
decisionmakers  and  the  assessment  team  itself.  Figure 
9.1 ,  Data  Taxonomy,  lays  out  a  number  of  the  types 
of  data  including  broad  categories  of  sources  that  will 
be  of  interest  to  the  analyst.  The  ability  to  determine 
the  needed  data  and  the  ability  to  assemble  or  collect 
this  data  determine  the  solution  strategy.  The  capability 
to  obtain  or  collect  the  appropriate  data: 

•  Will  often  reflect  the  difference  between  the  desired 
or  “ideal”  measures  of  merit  (MoM)  and  the  set  of 
MoM  actually  available  and  used  in  the  assessment; 
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•  Frequently  either  constrains  or  determines  the 
scenario  or  set  of  scenarios  that  are  used; 

•  Is  a  key  or  major  factor  in  determining  the  set  of 
tools  appropriate  for  the  assessment;  and 


•  Often  acts  as  a  schedule  and  cost  driver  for 
the  analysis. 


Common  Data  Infrastructure 

S 

o  . 


Official  Sources 

Legacy  Study 

Open  Sources 

|  (e  g  Gov  .  Mil  .  UN)  | 

Results 

(eg  WWW) 

Figure  9.1 .  Data  Taxonomy 


Reuse  of  Data 

Because  of  the  centrality  of  data  to  the  assessment, 
there  is  increasing  interest  in  data  reuse.  While  the 
amount  of  data  potentially  available  for  a  given  study 
is  growing  exponentially,  the  real  opportunities  for  the 
reuse  of  data  have  proven  to  be  limited  because: 
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•  The  rapid  change  of  technical  data  derailing 
performance  of  systems  or  sub-systems  must  be 
identified  by  version  and  data; 

•  Assessment  teams  do  not  always  know  that  data 
exist  and  have  no  easy  way  to  find  out  about 
such  “legacy”  data; 

•  The  data  that  are  available  are  seldom  in  an 
easily  accessible  form; 

•  The  conditions  under  which  the  data  were 
collected  are  not  documented; 

•  The  definitions,  languages,  and  measurement 
instruments  used  in  different  analyses  vary 
widely;  and 

•  Restrictions  arising  from  security  considerations. 

Despite  these  barriers,  an  effort  should  be  made  to 
find  and  reuse  data.  Sources  that  should  be  considered 
include  “official  sources”  such  as  the  customer,  open 
sources  such  as  those  available  on  the  Internet,  and 
prior  studies  on  similar  topics.  Because  the  data 
needed  will  seldom  be  available  in  a  form  and  format 
ideal  for  the  assessment,  data  engineering  is  often 
needed  to  gather,  organise,  and  transform  the 
available  data. 

Data  generated  by  a  given  assessment  phase  may,  in 
itself,  be  valuable  as  an  input  later  in  the  research  or 
to  other  project  teams.  To  enable  appropriate 
reusability  of  data,  every  modification,  constraint, 
assumption,  etc.  has  to  be  documented  adequately. 
To  facilitate  the  reuse  of  data,  it  is  best  practice  to  use 
metadata  for  this  purpose. 
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In  order  to  facilitate  the  reuse  of  data  across  community 
boundaries,  alignment  of  the  processes  and  methods 
for  data  and  metadata  modelling  on  the  mid  term — 
resulting  in  shareable  data  and  metadata  models  and 
a  common  ontology  on  the  long  term — is  necessary. 

Furthermore,  the  issue  of  verification,  validation,  and 
certification  (VV&C)  for  data  becomes  an  issue  of 
increasing  significance.  It  is  good  practise  to  use 
certified  data  whenever  possible. 

Definition  of  Data  Domains 

Data  can  be  directly  connected  to  the  other  sections 
of  the  Code  of  Best  Practice  (COBP)  by  respective 
data  domains.  These  data  domains  categorise  what 
the  information  is  about — and  what  data  are 
available — as  well  as  the  assessment  needs  of  the 
study — and  what  data  are  required.  Some  of  these 
categories  are: 

•  Scenario  data:  the  set  of  data  describing  the 
scenarios  and  vignettes; 

•  Human  organisational  issues  data:  the  set  of 
data  describing  the  scenarios  and  vignettes; 

•  System  performance  data:  the  set  of  data 
describing  system  performance  in  different 
scenarios;  and 

•  Tool  data:  the  set  of  additional  data  used  for  the 
tools  that  are  not  covered  by  any  other  category. 
Hard-wired  assumptions  belong  here  as  well  as 
study-specific  configuration  parameters  used  for 
technical  calibration  of  tools.  It  is  good  practice 
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to  be  aware  of  the  hidden  data  as  well  as  the 
input  parameters  for  each  tool  to  be  used  within 
the  study,  especially  when  it  is  planned  to  build 
federations  of  tools. 

Data  Sources 

Data  can  be  obtained  from  various  sources.  The  most 
common  sources  include: 

•  Official  sources:  sources  such  as  military 
databases,  governmental  data,  data  owned  by 
the  United  Nations,  etc.  The  customer,  or  other 
stakeholders,  controls  the  access  to  this  data  or 
is  at  least  aware  of  the  existence  and  structure  of 
the  data.  It  can  be  assumed  that  the  customer 
will  support  the  analyst  in  getting  access  to  the 
data  or  a  sanitised  version,  in  case  the  original 
data  are  not  releasable  to  the  study. 

•  Open  sources:  data  sources  that  are  neither 
influenced  nor  controlled  by  the  customer.  The 
Internet,  commercial  organisations,  as  well  as 
open  data  sources  of  non-participating 
organisations  are  examples. 

•  Legacy  study  results:  data  sources  derived  from 
other  studies  conducted  by  the  operations 
analysis  and  operations  research  (OA/OR) 
community,  including  political,  psychological,  and 
sociological  studies  and  Command,  Control, 
Communications,  Computers,  Intelligence, 
Surveillance,  and  Reconnaissance  (C4ISR)  tests 
and  evaluations.  They  may  be  the  delivered  as 
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the  result  of  a  former  study  as  well  as  having 
been  an  intermediate  result. 

If  needed  data  are  not  available  from  empirical  sources, 
subject  matter  experts  should  be  used  as  sources  to 
estimate  needed  values. 

Data  Classes 

Data  classes  describe  the  technical  aspects  of  datum 
format,  datum  types,  and  stage  of  processing.  Data  classes 
identified  in  the  context  of  this  section  are  the  following: 

•  Raw  data  are  unprocessed.  Raw  data  may  come 
from  observations  of  reality  or  artificial  reality,  the 
product  of  an  instrumented  reality,  experimental 
situations,  or  selected  artificial  realities. 

•  Processed  data  and  information  are  the  result  of 
transforming  one  or  more  raw  data  elements  into 
another  variable.  For  example,  one  or  more 
radar  returns  are  transformed  into  a  track 
(friendly  or  not). 

•  Aggregate  data  and  information  are  properties  of 
a  collection  of  elements.  For  example,  the 
movement  of  individuals  on  a  battlefield  versus 
the  movement  of  a  squad  or  a  platoon. 

•  Statistical  values1  are  assessed  on  a  sample  of  a 
population  and  characterised  by  this  population. 
These  include  mean,  modes,  medians,  standard 
deviations,  and  kurtosis.  Statistics  are  often  used 
as  parameters  in  assessments. 
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•  Derived  data  and  information  are  outputs  from  a 
formula  or  simulation  model  that  implicitly  incorporates 
a  set  of  assumptions.  For  example,  loss-exchange 
ratios  given  sensor  and  weapons  assumptions. 

•  Intermediate  data  and  information  are  the 
products  of  one  phase  or  component  of  the 
assessment  that  provides  input  to  another  phase 
or  component. 

Some  C2  assessments  use  assumptions  or  preset 
parameters  in  place  of  data  or  statistics. 

Use  of  Metadata 

Data  that  are  collected  without  adequate 
documentation  are  frequently  viewed  as  suspect.  To 
avoid  having  good  data  discarded  due  to  a  lack  of 
documentation,  acceptable  community  standards  for 
documentation  should  be  employed.  The  data  must 
be  clearly  described  in  a  manner  that  is  understandable 
to  a  subject  matter  expert,  not  directly  involved  with 
conducting  the  study.  The  data  description  must  be 
robust  enough  to  inspire  user  confidence  in  the  data. 
It  is  good  practice  to  record  these  considerations  in 
the  metadata  associated  with  the  data  to  facilitate  the 
data  use  and  reuse. 

Data  may  be  in  any  of  the  relevant  levels  of 
measurement — ratio,  interval,  ordinal,  or  nominal.  In  C2 
assessments,  some  significant  factors  may  be  nominal. 
It  is  good  practise  to  record  all  of  the  assumptions, 
constraints,  and  limitations  in  the  metadata. 

In  general,  every  time  data  are  modified  or  processed 
to  create  new  data,  appropriate  metadata 
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documentation  should  be  provided  to  ensure 
traceability  of  results,  validation,  verification,  and 
certification  of  respective  data,  and  reuse  of  data  in 
later  phases  or  within  other  studies.  This  also  allows 
dealing  with  the  challenge  of  multiple  instances  of  data 
in  different  studies. 

Data  and  Problem  Formulation 

The  initial  data  available  will  often  be  vague,  uncertain, 
incomplete,  and  contradictory.  Analysts  usually  prefer 
data  to  be  sharp,  certain,  complete,  and  consistent.  It 
is  necessary  to  be  explicit  about  the  assumptions 
inherent  in  this  transformation. 

If  the  assessment  team  finds  it  necessary  to  transform 
“soft”  data  to  “hard”  data  in  order  to  use  tools  that 
require  “hard”  data,  it  is  best  practice  to  record  the 
source,  perceptions  etc.  in  the  metadata  associated 
with  the  data. 

Obtaining  Data 

It  may  be  safely  assumed  that  not  all  of  the  required 
data  will  be  available  pre-packaged  for  the  study 
question.  Some  relevant  data  will  likely  be  submerged 
among  a  pool  of  irrelevant  information. 

Not  all  data  will  be  under  military/government  control. 
Data  belong  to  the  stakeholders  who  are  not 
necessarily  connected  to  (or  even  friendly  toward)  the 
customer  of  the  study.  A  lot  of  information  is  available 
from  open  sources.  The  challenge  is  to  find,  organise, 
verify,  process,  and  convert  it  into  the  data  needed. 
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The  team  should  be  aggressive  and  persistent  in  the 
pursuit  of  required  data. 

The  data  needed  will  seldom  be  available  in  its  raw 
form.  Often,  data  have  been  transformed  and 
aggregated.  When  tools  have  to  be  applied  to  derive 
data,  the  derived  data  should  be  tagged  with 
explanatory  metadata  to  record  that  information  in  a 
form  that  will  facilitate  its  reuse. 

If  the  data  are  not  available  and  can  neither  be 
aggregated  nor  derived  from  the  available  sources,  it 
is  good  practice  to  use  the  knowledge  of  subject  matter 
experts  to  generate  the  necessary  data.  Increasingly, 
when  C2  assessment  teams  are  tasked  with  exploring 
new  concepts  of  operation,  empirical  data  can  not  be 
expected  to  exist.  It  is  further  best  practice  to  document 
this  in  respective  metadata  and  replace  such 
assumption  based  data  as  early  as  possible  with 
empirical  data,  e.g.,  as  soon  as  another  study  delivers 
the  needed  inputs.  It  is  recommended  to  check  the 
respective  study  results  if  subjectively  generated  data 
are  replaced,  particularly  when  the  study  result  has 
been  shown  to  be  sensitive  to  this  data  element. 

The  team  needs  to  know: 

•  What  data  are  needed/structure  of  data; 

•  Preferred  data  (independent  variables  within 
the  MoM); 

•  Necessary  data  (to  be  able  to  drive  the  tools); 

•  Available  data  (derivation,  extraction, 
collection,  etc.); 


Who  owns  this  data; 
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•  Security  issues,  possibility  of  declassifying  or 
otherwise  obtaining  release  may  be  another  issue; 

•  Costs  to; 

•  Buy  data; 

•  Collect  data  (people,  time,  resources);  and 

•  Generate  data. 

It  is  essential  to  make  the  value  of  data  clear  to  all  levels 
of  decisionmakers  and  operational  planers  to  ensure 
that  data  collection  issues  are  included  in  all  phases  of 
the  study.  The  data  collection  and  engineering  plan,  as 
introduced  in  section  4-E-4,  as  part  of  the  solution 
strategies,  has  to  take  this  into  account. 

The  process  of  data  acquisition  is  important,  but  data 
acquisition  should  not  be  emphasised  over  data  interpretation. 

There  is  an  increasing  urgent  need  for  data  describing 
operations  over  the  complete  mission  spectrum.  It  is 
good  practise  to  collect  and  exploit  operational  data 
whenever  possible.  It  is  therefore  recommended  to 
synchronize  respective  data  collection  and  engineering 
plans,  ensuring  from  the  start  that  the  desired  data 
will  be  collected  and  archived  appropriately. 

The  Use  of  Data  within  the  Study 

The  archiving  of  data  in  retrievable  form  is  essential. 
This  is  necessary  both  to  support  the  ongoing  study 
and  also  to  be  of  value  for  future  study  efforts.  The 
team  should  establish  and  adopt  process  models  that 
ensure  the  build-up  of  archives  within  a  respective 
infrastructure.  Metadata  have  a  critical  role  to  play  in 
data  archiving  and  retrieval. 
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The  discussion  of  data,  and  the  development  of  a 
community  database,  must  be  driven  by  agreed  upon 
definitions  of  data  that  are  sufficient  to  support  the 
community.  This  should  be  a  high  priority  item. 

Data  can  be  described  as  the  “glue”  that  holds  together 
the  different  phases  of  a  study.  As  the  results  from 
one  phase  are  transported  to  the  next  it  is  unlikely 
that  both  will  use  identical  processes  and  procedures, 
therefore  harmonised  data  formats  will  allow  for 
smooth  transition  and  continuity  of  effort. 

The  analyst  will  be  faced  with  data  in  various  forms 
and  formats.  In  order  to  consider  all  available  data, 
find  it  if  necessary,  and  format  it  in  the  required  manner 
(e.g.,  as  an  input  parameter  for  the  MoM  or  a  model) 
data  must  be  stored  appropriately.  One  standard  for 
storing  data  is  the  Information  Resource  Dictionary 
Systems  (IRDS)  [ISO  1990]. 

IRDS  is  a  layered  database  that  not  only  comprises 
the  data,  but  also  the  metadata  describing  the  meaning 
of  the  data,  the  format,  the  constraints,  the  intended 
use,  the  source,  degree  of  certainty,  vagueness  and 
reliability,  etc.  It  comprises  data  as  well  as  metadata. 

A  common  understanding  of  the  problem  between  the 
customer  and  the  study  group  as  well  as  among  the 
interdisciplinary  study  team  is  essential  for  the  success 
of  the  study.  Therefore,  a  study  glossary — based  on  a 
general  and  evolving  OR  glossary  (e.g.,  the  NATO 
JOINT  Pub  1  -02  [US  DoD  1 999]) — is  needed.  The  data 
definitions  stored  as  metadata  have  to  be  aligned  with 
the  definitions  found  in  the  study  glossary.  This  provides 
a  basis  for  standardised  documentation  of  study  results 
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in  a  highly  reusable  form  that  can  be  manipulated  easily 
and  reliably,  perhaps  by  automated  systems. 

Using  the  right  toolkit  for  the  management  of  data 
within  the  IRDS,  such  as  the  Shared  Data  Environment 
(SHADE),  can  create  the  initial  state  of  every  future 
data  driven  application.  (DISA,  1996;  Krusche  and 
Tolk,  2000).  The  same  techniques  and  tools  can  and 
should  be  used  for  information  systems  delivering  the 
needed  functionality  to  the  warfighter  and 
decisionmaker.  (Tolk,  2000). 

Conclusions  and  Recommendations 

Data  transcend  all  phases  of  an  assessment  and  may 
be  seen  as  the  “glue”  that  holds  the  phases  of  an 
assessment  together.  Given  its  importance,  resources 
must  be  committed  to  ensure  effective  data  acquisition, 
management,  and  availability  for  reuse. 

There  is  an  urgent  need  to  agree  on  standards  for 
data,  metadata,  and  data  management,  including  the 
conditions  under  which  the  data  are  collected,  data 
element  definitions,  metrics,  etc.  It  is  good  practise  to 
use  established  standards,  where  appropriate,  such 
as  the  Source  for  Environmental  Representation  and 
Interchange  (SEDRIS™),  or  to  use  de  facto  standards 
like  Digital  Terrain  Elevation  Data  (DTED). 

As  the  data  being  used  today  by  the  analysts  will  be 
the  data  needed  tomorrow  by  systems  engineers, 
decisionmakers,  and  commanders  for  their  operations, 
alignment  of  the  standardisation  processes  and  the 
respective  toolsets  as  early  as  possible  with  the 
command  and  control  systems  community  is  good 
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practice.  A  significant  first  step  in  such  an  alignment 
would  be  using  the  same  IRDS  [NATO  1999].  A 
common  C4I  and  Modelling  and  Simulation  (M&S) 
community  is  needed,  to  make  visions  like  integrated 
alternative  course  of  action  analyses  become  a  reality. 

Chapter  9  Acronyms 

C2  -  Command  and  Control 

COBP  -  Code  of  Best  Practice 

C4ISR- Command,  Control,  Communications,  Computers, 
Intelligence,  Surveillance,  and  Reconnaissance 

IRDS  -  Information  Resource  Dictionary  Systems 

MoM  -  Measures  of  Merit 

M&S  -  Modelling  and  Simulation 

OA/OR  -  Operations  Analysis  and  Operations  Research 

SHADE  -  Shared  Data  Environment 
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Risk  and 
Uncertainty 


We  may  at  once  admit  that  any 
inference  from  the  particular  to  the 
general  must  be  attended  with  some 
degree  of  uncertainty,  but  this  is  not 
the  same  as  to  admit  that  such 
inference  cannot  be  absolutely 
rigorous,  for  the  nature  and  degree  of 
the  uncertainty  may  itself  be  capable 
of  rigorous  expression. 

— R.A.  Fisher,  The  Design  of  Experiments 
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There  are  risks  associated  with  the  decisionmaker’s 
situation  that  are  an  inherent  part  of  the  analysis. 
There  are  also  risks  related  to  the  conduct  of  the 
analysis  itself.  This  chapter  deals  with  both  of  these 
sets  of  risks,  focusing  upon  reducing  uncertainty  and 
other  contributors  to  risk  as  well  as  the  mitigation  of 
their  effects.  Failure  to  deal  effectively  with  risks  will 
jeopardise  the  accomplishment  of  the  goals  of  the 
study,  namely  to  provide  high  quality  decision  support. 
It  should  be  noted  that,  by  the  end  of  the  assessment, 
the  team  will  generally  be  aware  that  the  initial  study 
plan  will  have  been  changed,  whereas  flaws  in  the 
conceptual  model  or  in  the  assessment  design  may 
remain  hidden  for  some  time  from  both  the  analyst 
and  the  decisionmaker. 

While  the  adoption  of  the  guidance  contained  in  this 
Code  of  Best  Practice  (COBP)  will  help  minimise  the 
risks,  it  will  not  eliminate  them.  This  chapter  discusses 
a  number  of  issues  related  to  the  risks  and 
uncertainties  that  the  assessment  team  needs  to 
explicitly  address. 

Risk 

Risk  is  commonly  defined  as  the  possibility  of  suffering 
harm  or  loss;  in  other  words,  an  exposure  to  harm  or 
loss.1  This  includes  an  opportunity  loss.  The  difference 
between  risk  and  loss  is  that  risk  inherently  involves 
before  the  fact  probability,  while  loss  is  an  after  the 
fact  certainty.  Insurance  is  about  estimating  and 
covering  an  exposure  of  value  to  uncertainty.  Risk  often 
has  a  negative  connotation,  yet  “taking  risks”  can  also 
be  a  positive  act  when  a  proper  balance  or  trade-off  is 
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made  between  good  and  bad  outcomes  and  the 
respective  probabilities  associated  with  the  outcomes. 

Perceptions  of  risk  can  substantially  differ  from  more 
objective  assessments  of  probability  and  impact.  This 
is  particularly  important  in  command  and  control  (C2) 
problems,  which  have  a  high  sociological  content. 

Uncertainty 

Uncertainty  can  be  generally  defined  as  an  inability  to 
determine  a  variable  value  or  system  state  (or  nature) 
or  to  predict  its  future  evolution.  Uncertainty  is  inherent 
in  risk.  Even  if  a  person  is  certain  about  the  possible 
outcomes  and  the  probabilities  associated  with  these 
outcomes,  there  is  uncertainty  about  which  outcome 
will  in  fact  occur.  When  this  situation  applies  we  can 
say  that  we  have  a  known  risk.  For  example,  the 
possible  outcomes  and  the  probabilities  associated 
with  the  toss  of  a  coin  (and  many  other  popular  forms 
of  gambling)  are  known.  What  is  uncertain  is  the 
outcome.  Hence  there  is  a  risk,  in  this  case  a  known 
risk,  associated  with  obtaining  a  particular  outcome. 

There  are  other  types  of  uncertainty,  including 
uncertainty  about  the  possible  outcomes  (or  their 
values)  and  uncertainty  about  the  probabilities 
associated  with  these  outcomes.  Uncertainties  arise 
from  other  uncertainties,  namely  uncertainties  about 
the  potential  actions  of  others  or  what  is  referred  to  as 
“states  of  nature”  and  their  associated  probabilities. 
In  C2-related  assessments,  in  particular,  perceptions 
of  uncertainty  may  substantially  differ  from  a  more 
objective  assessment  of  lack  of  knowledge.  There  are 
also  a  number  of  uncertainties  that  are  associated  with 
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the  assessment  itself.  These  are  discussed  later  in 
this  chapter. 

Dealing  with  Risk 

There  are  three  basic  ways  to  deal  with  risk.  The  first  is 
to  reduce  the  uncertainty  that  underlies  the  risk.  The 
second  is  to  mitigate  risk  by  developing  and  selecting 
risk-averse  strategies.  The  third  is  to  effectively 
communicate  the  nature  of  the  risk  involved. 

Reducing  Uncertainties 

In  effect,  all  methods  for  reducing  the  uncertainties 
that  underlie  risk  involve  the  collection  or  analysis  of 
information.  This  is,  of  course,  a  major  focus  of  an 
assessment.  Clearly  the  assessment  team  needs  to 
identify  the  most  important  uncertainties — a 
combination  of  the  degree  of  uncertainty  and  the 
consequences  of  that  uncertainty.  However,  there  will 
always  be  some  significant  residual  uncertainties.  The 
aim  of  the  assessment  team,  at  this  point,  is  to  reduce 
the  risk  associated  with  the  study  by  learning  more 
about  the  robustness  (or  lack  thereof)  of  the  study 
findings  and  conclusion.  Sensitivity  analysis  is  one 
method  to  accomplish  this  purpose. 

Sensitivity  Analysis 

The  goals  of  sensitivity  analysis  are  fourfold: 

•  To  establish  the  regions  for  which  established 
results  are  valid; 
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•  To  isolate  those  factors  that  contribute  most  to 
the  uncertainties  that  exist  and  identify  the  risks 
associated  with  these  uncertainties  as  they 
relate  to  study  findings  and  conclusions; 

•  To  make  study  results  more  robust  for  decision 
support  by  allowing  the  decisionmaker  to  see  the 
immediate  consequences  as  external  factors  are 
changing;  and 

•  To  give  the  decisionmaker  a  richer  understanding 
of  the  decision  problem,  highlighting  the 
consequences  of  limited  changes  relative  to  a 
solution  proposed  by  the  study. 

The  team  should  take  a  three-step  approach: 

•  Identify  the  variables  that  are  associated  with  the 
greatest  combination  of  risk  and  uncertainty; 

•  Perform  sensitivity  analyses  that  vary  across  the  more 
likely  region  of  key  parameters  (often  the 
neighbourhood  of  the  initial  estimate)  to  see  how  the 
result  is  influenced.  A  sensitivity  analysis  investigates 
the  main  region  of  the  output  space;  and 

•  Investigate  the  extremes  of  the  same  output  space, 
focusing  most  strongly  on  the  negative  regions. 

The  assessment  team  also  partially  takes  an  inverted 
approach,  looking  for  possible  failures,  and  then 
seeking  possible  sources  of  each  kind  of  failure. 

Together,  these  three  steps  should  provide  the  team 
with  a  better  understanding  of  residual  uncertainties 
and  associated  risks.  The  need  for  and  results  of 
sensitivity  analyses  must  be  stressed  in  discussions 
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with  the  decisionmakers.  This  will  help  avoid  the  under¬ 
resourcing  of  this  critical  activity. 

Uncertainty  and  Risk  in  C2 
Assessments 

Uncertainty  and  the  risks  associated  with  it  can  not  be 
totally  eliminated  in  any  real  world  C2  assessment 
problem.  Moreover,  in  most  real-life  problems,  one 
cannot  even  identify  all  of  the  unknowns.  This  has 
given  rise  to  the  term  the  “unknown  unknowns.” 
Strategies  to  minimise  risk  have  evolved  to  handle 
these  facts  of  life.  Therefore,  even  the  best  possible 
assessment  approach  will  result  in  residual  uncertainty 
and  risks. 

In  general,  assessments  should  be  judged  by  their 
ability  to  reduce  uncertainty  so  that  the  decisionmaker 
is  in  a  better  position  (less  risk)  after  the  assessment 
than  before  the  assessment.  Looking  at  the  absolute 
uncertainty  that  remains  is  not  as  useful  a  measure 
since  it  may  say  more  about  the  nature  of  the  problem 
than  the  success  of  the  assessment. 

Dealing  with  Uncertainty 

It  is  important  to  treat  uncertainty  consistently  and 
explicitly.  This  allows  information  from  two  given 
sources  or  results  to  be  fused,  (e.g.,  by  taking  the  most 
precise  assessment  of  each  factor  from  the  two 
results).  Thus  the  resulting  knowledge  will  be  better 
than  either  of  the  two  separate  results.  On  the  other 
hand,  if  uncertainty  is  not  treated  explicitly  or 
consistently,  the  best  one  can  do  is  to  pick  the  single 
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result  that  seems  best.  This  makes  it  more  difficult  for 
a  study  to  add  value  to  a  decisionmaker. 

C2  issues  are  complex,  and  it  is  an  understatement  to 
say  they  are  incompletely  understood.  C2 
requirements  and  solutions  tend  to  depend  heavily  on 
the  nature  of  the  operating  environment.  Thus,  a  C2- 
related  study  will  rely  on  factors  that  are  imprecisely 
determined  and  change  frequently  over  time.  Almost 
regardless  of  the  assessment  effort,  parts  of  the 
problem-space  will  have  been  investigated  less 
thoroughly  than  would  be  ideal.  Sensitivity  analyses 
are  required  to  give  high  quality  study  results. 

C2  assessment  problems  (particularly  in  operations  other 
than  war  (OOTW)  contexts)  generally  have  many 
interacting  factors.  It  is  unwise  to  rely  upon  single  factor 
sensitivity  analysis  (i.e.,  testing  sensitivity  to  one  factor 
at  a  time).  Multi-factorial  experimental  design  methods 
are  good  practice  in  such  circumstances.  (Keppel,  1 973).2 

OOTW  studies  typically  have  less  well-formed 
quantitative  factors  and  more  qualitative  factors.  The 
“softer”  nature  of  these  factors  makes  assessment 
more  difficult.  The  types  of  factor  seen  in  OOTW 
problems  include: 

•  Social  and  political  activity  impacting  the  tactical  level; 

•  A  strong  influence  of  negotiation  and  persuasion 
as  opposed  to  coercion; 

•  Non-optimal  performance  of  military  capabilities 
from  a  technical  perspective  due  to  their  poor  fit 
to  the  problem; 

•  Severe  rules  of  engagement  (ROE)  constraints;  and 
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•  Unclear  or  evolving  goals  and  objectives. 

The  team  must  be  aware  of  the  assumptions  and 
limitations  included  in  models,  scenarios,  and  data 
structures,  which  should  be  captured  in  the  metadata. 
In  particular,  it  should  be  noted  that  humans  involved 
in  C2  experiments  (as  analysts  or  subjects)  always 
bring  assumptions  with  them.  These  need  to  be 
identified  and  collected  to  form  an  audit  trail.  This  is 
one  place  where  the  project  leader’s  log  of 
assumptions  and  decisions  made  during  the 
assessment  pays  off.  Uncertainty  over  the  validity  of 
these  assumptions  and  limitations  provide  a  source 
of  uncertainty  in  study  results  to  which  study 
conclusions  must  be  made  robust. 

In  C2  assessments,  in  particular,  all  aspects  of  a  study 
and  study-problem  may  be  connected  to  uncertainty. 
Thus,  different  sorts  of  uncertainty  should  be 
addressed  explicitly  at  appropriate  stages  in  the  study. 
Examples  of  these  are: 

•  Parameter  value  uncertainty — many  of  the 
parameters  and  factors  in  C2  assessment  are 
difficult  to  evaluate  confidently; 

•  Model-based  uncertainty — i.e.,  over  whether 
underlying  models  are  valid  and  representative; 

•  Uncertainty  of  focus — i.e.,  over  whether  the 
assessment  covers  all  of  the  important  factors 
and/or  issues  (this  includes  uncertainty  of 
scenarios);  and 

•  Complexity  of  uncertain  factors  (i.e.,  their 
dimensionality) — when  a  sufficiently  complex 
factor  (e.g.,  scenarios  or  future  technology)  is 
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uncertain,  the  team  can  not  expect  to  overview 
the  set  of  ail  possible  true  states. 

Uncertainty  in  complex  factors,  such  as  scenarios, 
should  be  addressed  thoroughly.  Even  though  it  is 
philosophically  impossible  to  know  everything  about 
a  problem,  an  adequately  complete  knowledge  can 
be  better  assured  by  explicit  use  of  checklists  that 
highlight  the  breadth  of  factors  typically  involved  in  C2 
assessments,  such  as: 

•  Technology — disruptive  uncertainty  and 
disruptive  innovation; 

•  Organisational  use  of  technology; 

•  Scenarios,  tasks,  and  nature  of  operations; 

•  Data; 

•  Context  or  environment  of  the  assessment;  and 

•  Co-evolution  of  factors  (e.g.,  summarised  in 
doctrine,  organisation,  training,  material, 
leadership,  personnel,  and  facilities 
[DOTMLPF]). 

Nonetheless,  the  team  should  not  rely  completely  on 
checklists,  but  rather  complement  them  with  critical 
thinking  in  the  specific  study  context. 

Within  electronic  systems,  organisations,  and  battle 
concepts,  there  is  a  lot  of  opportunity  for  disruptive 
technology,  producing  substantial  uncertainty.  In 
human-in-the-loop  experiments  there  is  a  greater 
danger  of  bias  in  subjective  judgements.3 
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In  considering  sensitivity  analysis,  it  is  important  not 
to  associate  it  only  with  statistical  variance  in  parameter 
values.  Qualitative  consideration,  such  as  variation  of 
model,  perspective,  or  assumptions  (i.e.,  categorical 
variations)  should  also  be  used  to  test  for  and  assess 
sensitivity.  Variations  in  ranking  (ordinal  variation)  can 
also  be  a  powerful  tool  for  sensitivity  analysis.  Other 
analytic  tools  and  constructs  relevant  to  sensitivity 
analysis  are: 

•  Non-parametric  statistics  (Siegal  and  Castellan,  1 988); 

•  Belief-functions  (as  an  alternative  to  probability); 

•  Judgmental  uncertainties  (Wilson  and  Corlett,  1 990); 

•  Fuzzy  numbers,  theories  of  semiorders,  scoring 
criteria  (Siegal  and  Castellan,  1 998); 

•  Multi-dimensional  scaling;  and 

•  Mathematics  applied  to  non-ordinal  scales. 

Semiorders  constitute  the  intermediate  level  between 
ordinal  information  and  value.  They  apply  to  many 
fields  but  in  this  context  especially  to  scored  data  or 
preference  data — in  that  account  is  taken  of  the 
intervals  of  imprecision  around  the  measuring  systems 
used.  Analysts  should  be  aware  of  the  thresholds 
associated  with  the  collected  data  above  which 
differences  can  be  legitimately  distinguished  to 
produce  values.  This  idea  is  very  useful  when  the 
measurements  cannot  be  repeated  (as  in  statistical 
theory)  (Prilot  and  Vincke,  1997). 

Keeping  C2  assessment  rigorous  and  robust  in  the 
face  of  the  many  uncertainties  and  complexities  of  the 
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subject  matter,  as  well  as  the  need  to  use  a  rich 
combination  of  methods,  can  be  difficult.  Again,  it  is 
good  practice  to  use  checklists  and  structured 
appraisal,  in  this  case  to  maintain  an  objective  view  of 
study  rigour.  The  choice  of  checklist  or  appraisal 
structure  can  depend  upon  personal  preferences,  but 
Annex  F  lists  a  number  of  structures,  which  have 
proved  useful  in  the  experience  of  the  nations 
contributing  to  this  COBP.  These  include: 

•  Repeatability,  independence,  grounding  in  reality, 
objectivity  of  process,  uncertainty  and 
robustness  (RIGOUR);  and 

•  Strengths,  weaknesses,  opportunities,  and 
threats  (SWOT). 

Where  an  assessment  uses  experimentation  or 
observation  of  exercises  it  is  important  to  identify 
independent  and  dependent  variables  and,  for  the 
former,  which  are  controllable,  which  are  measurable, 
etc.  Figure  10.1,  for  example,  illustrates  the  variety  of 
variables  that  need  to  be  considered  when  studying 
decisionmaking. 

A  thorough  understanding  of  the  variables  of  a  study 
is  essential  for  effective  treatment  for  uncertainty. 
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INPUTS 


Personal  Institutional 

-  Culture  •  Culture 

-  Style  -  Doctrine 


-  etc.  -  etc. 

CONSTRAINTS 

4  ► 


OUTPUTS 


•  Sensor"  Information — 
(with  variability) 


Decision  Making 

- 3* -  - *  v 


Action  Instructions 
(with  variability) 


RESOURCES 


Personal 

-  memory 

-  experience/knowleOge 

-  skill  expertise 
-etc. 


Institutional 

-  memory 

-  Information  technology 

-  administrative  support 
etc. 


Figure.  1 0.1 .  Illustration  of  the  Variety  of  Variables 
Relevant  to  Decisionmaking 


Risk-Based  Analysis 

In  cost-benefit  or  cost-effectiveness  analyses  there  has 
historically  been  a  tendency  to  focus  on  the  single  most 
likely  value  (its  expected  value)  for  each  input  factor 
(including  scenario  and  course  of  action)  and  to  “solve” 
the  problem  based  upon  these  point  estimates.  This 
can  lead  to  fragile  solutions,  which  do  not  provide 
decisionmakers  with  help  in  dealing  with  the 
uncertainties  and  associated  risks  inherent  in  the  real 
problem.  A  risk-based  approach  can  overcome  some 
major  pitfalls  by  adding  a  focus  on  the  multiplicity  of 
possible  outcomes  and  opening  up  the  possibility  of 
richer  solutions  involving  portfolios  of  action  that  produce 
robustness  rather  than  narrow  optimality.  Portfolio- 
based  solutions  can  be  associated  with  cost-benefit 
approaches,  but  this  has  not  been  common  in  practice. 

The  subject  of  C2  assessments  is  typically  both  highly 
uncertain  and  opaque  in  nature.  The  team  should, 
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therefore,  expect  a  complex  and  partly  hidden  set  of 
risks  to  C2  studies  and  C2-related  decisions.  It  is 
recommended  that  serious  efforts  be  made  to  illuminate 
the  risks,  and  it  is  good  practice  to  include  an  explicit 
risk-based  analysis  in  the  study  and  in  study  planning. 

Different  people  have  different  worldviews  and  different 
approaches  to  risk  taking.  Analysts  should  seek  to  find 
out  about  how  risks  are  traded  for  expected  gain  by 
the  study  sponsor  so  that  their  worldview  can  be 
appropriately  represented  in  the  assessment. 

Risk-based  analysis  needs  metrics  for  risks  and  failure 
as  well  as  success  and  benefits,  which  means  that 
one  needs  a  way  of  expressing  various  levels  of  all  of 
these  dimensions. 

In  C2  assessments,  analysts  need  to  be  particularly 
alert  to  the  possibility  of  chaotic  behaviours  arising  from 
dynamic  interactions.  Human  and  organisational 
factors  are  particularly  prone  to  this  type  of  instability. 
A  sound  and  explicit  treatment  of  boundaries  and 
system  definitions  during  problem  formulation  is 
essential  to  managing  this  aspect  of  the  assessment. 
Holistic  systems  thinking  and  complexity-based 
analysis  may  be  needed  for  this  purpose. 

When  dealing  with  problems  involving  human 
decisionmaking  the  analyst  must  be  aware  of  the 
diversity  of  courses  of  action  that  are  possible  as  the 
situation  evolves.  The  analyst  must  ensure  that  these 
courses  of  action  are  represented  in  a  way  that  allows 
for  the  possibility  of  a  discontinuous  set  of  possible 
future  states.  If  wide  divergence  in  course  of  action 
selection  is  possible  over  the  timescales  under 
consideration,  then  the  treatment  of  scenarios  may 
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be  particularly  problematic  and  may  require  explicit 
consideration  under  the  risk  and  sensitivity  heading. 

Managing  Study  Risk 

C2  assessments  are  inherently  complex.  They  often 
contain  poorly  understood  study  problems.  These 
factors  enhance  the  level  of  risk  in  the  design  and 
conduct  of  the  assessment. 

It  is  good  practice  to  try  to  make  a  complete  list  of 
risks  and  then  treat  them  in  appropriate  detail.  A  top- 
down  approach  may  be  useful  in  assuring  a  certain 
degree  of  completeness.  Independently  of  the  depth 
of  each  concrete  study,  it  is  good  practice  to  use  a 
risk  perspective  to  explicitly  assess  the  robustness  of 
a  conclusion  or  recommendation.  If  possible,  one 
should  try  to  keep  track  of  which  mechanisms  underlie 
each  risk,  the  probability  that  it  will  occur  and  how  it 
can  be  mitigated,  then  consider  cost  of  mitigation  and 
cost  of  risk  impact,  before  taking  a  cost-benefit 
approach  in  managing  the  study  risk  level. 

C2  problems  are  often  weakly  bounded.  There  is  a 
particular  risk  associated  with  problem  formulation  and 
the  identification  of  factors.  Annex  F  lists  a  number  of 
checklists,  which  have  proved  useful  in  ensuring  an 
adequately  complete  treatment  of  the  multiple  risk  areas. 

The  Generic  Risk  Register 

The  generic  risk  register  for  C2  assessment  (GRR)  is 
a  good  starting  point  for  a  risk-based  analysis  of  a 
study  project  (not  the  decision  problem  supported  by 
the  study).  The  lists  of  risks,  consequences  of  impact, 
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and  mitigation  recommendations  are  directly  derived 
from  the  COBP.  They  are,  therefore,  generic  in  their 
form  and  should  only  be  taken  as  a  starting  point  for  a 
project-specific  risk  analysis.  The  GRR  has 
functionality,  which  allows  it  to  be  used  as  a  basic  tool, 
or  the  list  of  risks  can  be  copied  into  a  more  elaborate 
tool  used  for  risk  handling  in  the  project. 

An  illustrative  example  is  the  case  study  undertaken 
by  the  SAS-026  study  group,  when  a  brief  journey  of 
only  an  hour  through  the  generic  risk  register  turned 
out  very  useful.  Although  the  study  team  was  well 
aware  of  the  advice  and  possible  pitfalls  in  advance, 
explicitly  addressing  them  with  a  risk  perspective  led 
to  the  recognition  of  two  significant  flaws: 

•  The  low  number  of  planned  iteration  had  the 
potential  to  lead  to  a  risk  of  an  inefficient  and 
unfocused  study  with  possibly  misleading 
results;  and 

•  The  relatively  narrow  selection  of  methodological 
approaches  entailed  a  risk  of  misleading 
conclusions.  There  could  be  important 
consequences  of  varying  the  C2  system  that 
were  not  reflected  in  the  study,  and  the  possibly 
biased  representation  would  represent  a  hidden 
flaw  in  conclusions. 

The  example  illustrated  both  the  usefulness  of  making 
the  risk  assessment  explicit,  since  these  problems  were 
actually  known  to  all  participants  prior  to  the  risk 
management  session,  and  also  illustrated  that  even 
surface  scratching  (as  was  the  case  here)  may  lead  to 
significant  results.  It  is,  therefore,  advisable  not  to  skip 
risk  analysis  even  when  time  and  resources  are  limited. 
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Communication  of  Risk  and  Uncertainty 

The  high  level  of  uncertainty  (and  hence  risk)  in  C2 
problems  and  their  assessment  mean  that  the 
communication  of  risk  and  uncertainty  to  study 
customers,  sponsors,  and  stakeholders  is  of  particular 
importance.  The  value  of  a  quality  assessment  is  that  it 
provides  decisionmakers  with  the  evidence  they  need  to 
make  better  decisions.  The  nature  and  quality  of  evidence 
required  depend  upon  the  decisionmaker’s  approach  to 
and  tolerance  for  risk-taking  and  his/her  level  of  prior 
knowledge  of  the  problem  area  being  assessed. 

Communication  is  about  giving  the  receiver  of  a 
message  a  right  impression,  not  about  formulating  a 
statement  that  is  formally  correct  on  its  own.  This  might 
seem  obvious,  but  in  communicating  uncertainties  and 
risk,  it  should  be  given  particular  attention,  since  the 
complexities  of  the  subject,  human  limitations  in 
reflecting  on  uncertainty,  and  the  lack  of  a  common 
set  of  concepts  (and  also  hidden  agendas)  often  will 
make  communication  far  less  than  perfect. 

As  discussed  earlier,  some  uncertainty  can  be  reduced 
by  analysis.  However,  some  uncertainty  is  inherent  in 
the  problem  and  needs  to  be  exposed  to  the 
decisionmaker.  A  failure  to  do  this  can  lead  to  false 
confidence  in  study  conclusions.  C2  assessments,  in 
particular,  will  contain  many  areas  of  unresolvable 
doubt  and  uncertainty.  These  should  be  openly  and 
honestly  communicated  to  decisionmakers  to  avoid 
misinterpretation  of  study  conclusions.  Support  to 
decisionmaking  under  uncertainty  is  a  vital 
complementary  activity  to  C2  assessment. 
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Different  ways  of  framing  results  and  uncertainties  may 
strongly  influence  the  way  results  are  perceived.  This 
should  be  considered  thoroughly  to  assure  compliance 
with  ethical  standards.  One  should  be  aware  that 
stakeholders  (including  customers)  might  have  atendency 
to  gloss  over  or  alternately  over-focus  on  uncertainties. 
An  analyst  should  take  care  in  communicating  an  objective 
impression  of  risks  and  uncertainty. 

Limitations  and  shortcomings  in  a  study  are  a  crucial 
part  of  the  study  result  and  should  be  communicated  as 
effectively  as  possible.  This  enables  alignment  of  study 
results  and  background  knowledge  on  the  problem. 

Human  ability  to  understand  and  reason  on  uncertainty 
is  limited.  (Kahnmann  et  al.,  1 982).  These  limitations 
should  be  given  particular  attention  when 
communicating  risk  and  uncertainty  to  stakeholders 
and  decisionmakers. 

People  with  different  backgrounds  will  have  different 
concepts  of  uncertainty  (e.g.,  people  without  some 
mathematical  background  won’t  necessarily  intuitively 
understand  Bayesian  concepts).  Thorough  dialogue 
may  be  needed  to  find  a  common  language. 
Visualisation  techniques  will  be  helpful  in  this  regard 
since  they  are  usually  more  powerful  than  verbal 
reference  to  abstract  concepts. 

One  should  be  careful  not  to  overwhelm  an  audience 
with  details  on  uncertainties  and  possible 
shortcomings.  However,  a  continuing  dialogue  about 
uncertainty  will  facilitate  a  common  understanding. 
Also,  the  analysis  team  should  be  aware  of  the 
possibility  that  residual  uncertainties  may  make  it 
impossible  to  draw  robust  conclusions. 
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Conclusions 

The  explicit  treatment  of  risk  and  uncertainty  is  best 
practice  in  all  studies  and  is  of  particular  importance 
in  C2  assessment.  A  variety  of  dimensions  and  aspects 
of  C2  assessments  carry  risk  and  uncertainty, 
particularly  because  they  are  liable  to  include  complex, 
interacting  factors.  Even  when  study  resources  are 
limited,  it  is  best  practice  to  include  both  an  assessment 
of  most  likely  outcome  (result),  to  do  sensitivity 
analyses  looking  for  other  likely  outcomes,  and  to  take 
a  risk-based  approach  looking  for  the  more  extreme 
possible  outcomes  (in  particular  failures). 

The  use  of  checklists  is  recommended  to  ensure  a 
rigorous  treatment  of  risk  and  uncertainty.  The  best 
choice  of  checklist  depends  upon  personal  preference, 
but  a  number  of  examples  are  presented  that  have 
been  found  useful  by  the  nations  contributing  to  this 
COBP.  Additionally,  the  GRR  has  proved  useful  as  an 
aid  to  C2  study  risk  management. 

Chapter  10  Acronyms 

C2  -  Command  and  Control 

COBP  -  Code  of  Best  Practice 

DOTMLP-  Doctrine,  Organisation,  Training,  Material, 
Logistics,  Personnel 

DOTMLPF  -  Doctrine,  Organisation,  T raining,  Material, 
Leadership,  Personnel,  Facilities 

GRR  -  General  Risk  Register 
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METT-TC  -  Mission,  Enemy,  Troops,  Terrain,  Troops, 
Time,  and  Civil  considerations 

OOTW  -  Operations  Other  Than  War 

PESTLE  -  Political,  Economic,  Social,  Technological, 
Legal,  and  Environmental 

RIGOUR  -  Repeatability,  Independence,  Grounding  in 
reality,  Objectivity  of  process,  Uncertainty,  and  Robustness 

ROE  -  Rules  of  Engagement 

SWOT  -Strengths,  Weaknesses,  Opportunities,  Threats 
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'This  interpretation  of  risk  is  used  throughout  this  section. 
However,  other  common  definitions  have  been  included  at  Annex 
E  as  an  aid  to  common  understanding. 

2Tom  Lucas’  work  in  Project  Albert. 

3These  types  of  experiment  are  typically  unrepeatable  due  to 
resource  constraints. 
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Products 


We  are  what  we  repeatedly  do. 

Excellence,  then,  is  not  an  act,  but 
a  habit. 

— Aristotle 

The  purpose  of  study  products  is  threefold:  to 
communicate  results  to  sponsors  and 
stakeholders;  to  provide  a  lasting  record  of  what  went 
into  the  planning;  and  to  establish  credibility  within  the 
technical  community.  Verbal  communication  and 
progress  reports  may  be  necessary,  especially  for  a 
short  study,  however  a  thorough  written  record  is 
essential  to  the  credibility  and  longevity  of  study  results. 
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Study  products  that  are  delivered  to  the  customer 
generally  include  a  study  plan,  periodic  status/progress 
reports,  and  a  final  report.  Several  other  products  may 
be  produced  and  delivered  to  the  customer.  Some 
study  products  are  created  and  maintained  primarily 
for  internal  study  support.  These  products  are  not 
unique  to  C2  OOTW  studies  or  to  general  C2  studies. 

Products  that  are  typically  produced  from  a  study  include: 

Study  Plan 

The  study  plan  described  here  is  a  subset  of  the  Study 
Management  Plan  of  section  4-E.  It  generally  includes 
the  initial  problem  formulation  and  solution  strategy, 
emphasizing  a  general  understanding  of  the  problem, 
deliverables,  budget,  time  line,  and  solution  approach. 
The  study  plan  includes  at  a  minimum: 

•  Statement  of  the  problem  (problem 
formulation) — “the  what”; 

•  Solution  strategy  “the  how  and  the  when”; 

•  Tasks  and  their  relationships;  and 

•  Milestones. 

Periodic  Status/Progress  Reports 

Periodic  status  reports  describe  the  activities  of  the  most 
recent  period  and  the  expected  activities  of  the  next 
period,  and  link  the  activities  to  the  tasking  statement. 
Status  reports  also  contain  cost  information  and  track 
adherence  to  the  planned  schedule.  One  of  the  most 
important  sections  of  the  status  report  is  the  “Problems 
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Encountered”  section.  This  section  should  include 
technical  problems,  budgetary  problems,  and  most 
importantly  problems  relating  to  sponsor/assessment 
team  relations.  These  reports  may  be  delivered: 

•  To  the  sponsor  and  other  stakeholders;  and 

•  To  the  peer  review  team. 

Final  Report 

The  final  report  contains  sections  that  address  the  following : 

•  Objectives  (customer  question  and  the 
problem  formulation); 

•  Scope  and  assumptions; 

•  Approach  (solution  strategy); 

•  Findings/conclusions  (with  caveats); 

•  Recommendations  (optional); 

•  Future  challenges  (optional); 

•  Appendices; 

•  Data  collection  instruments  or  discussion  of 
instrumentation  (optional); 

•  Data  dictionary  (optional); 

•  Data  (optional); 

•  Glossary  of  terms/acronyms;  and 

•  References. 


270  NATO  Code  of  Best  Practice  for  C2  Assessment 


The  findings/conclusions  and  recommendations  (if 
present)  should  address  each  of  the  objectives. 

The  final  report  should  be  produced  as  an  archivable 
document  that  can  be  readily  accessed  by  the 
community.  This  means  that  the  document  is  produced 
in  electronic  form  using  commercial  standards.  Ideally 
these  documents  should  be  made  available  through 
web  sites.  However,  it  is  recognised  that  security 
considerations  and  language  will  limit  the  availability 
of  many  documents.  Briefings  are  a  useful  form  of 
communication.  It  is  desirable  that  a  briefing 
accompany  the  more  formal  final  report.  However, 
when  only  a  briefing  is  produced,  it  must  be  annotated. 

Command  and  control  (C2)  problems  tend  to  be  multi¬ 
dimensional  and  highly  complex.  Therefore,  they  pose 
unique  challenges  to  the  assessment  team  in 
communicating  the  result  effectively  to  the 
decisionmaker.  There  are  several  steps  that  can  be 
taken  to  facilitate  this  communication.  First,  if  the  product 
is  an  architecture,  it  is  recommended  that  the  templates 
developed  by  community  to  depict  alternative 
architectural  perspectives  (e.g.,  operational,  system, 
and  technical  architectures)  be  employed.  Second,  it 
has  proven  useful  to  employ  “stop  light”  charts  (red, 
yellow,  and  green)  to  characterise  measures  of 
performance/effectiveness.  However,  such  techniques 
are  useful  only  in  conveying  qualitative  insights.  A 
tendency  to  rely  heavily  upon  this  presentation 
technique  may  result  in  assessments  that  do  not  drive 
to  quantitative  results.  Third,  there  is  a  need  to  develop 
and  employ  visualisation  techniques  to  capture  the  full 
richness  of  the  insights,  particularly  risk  and  uncertainty 
(e.g.,  depicts  the  distribution  rather  than  just  the 
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statistical)  that  are  derived  in  assessments.  Preliminary 
research  is  underway  in  this  area.  It  needs  to  be 
extended  and  translated  into  application. 

Because  C2  data  are  rare,  every  effort  should  be  made 
to  retain  the  data  and  make  it  available  to  other 
recorders.  Sometimes  others  will  require  “sanitising” 
the  data  to  prevent  anyone  knowing  which  units  or 
exercises  produced  it.  When  archiving  data  the 
metadata  labels  that  identify  the  conditions  under  which 
it  was  collected  should  also  be  preserved. 

A  peer  review  process  is  an  essential  part  of  producing 
a  final  report.  It  should  begin  with  the  preparation  of 
the  study  plan  and  continue  throughout  the  life  of  the 
assessment.  Draft  products  must  be  provided  to 
reviewers  in  ample  time  for  them  to  review  and 
comment  on  the  product  and  for  the  team  to  reflect 
their  comments.  A  failure  to  institute  an  adequate 
review  process  can  compromise  the  quality,  credibility, 
and  utility  of  the  assessment. 

Other  Delivered  Products 

Several  products  may  be  created  and  delivered, 
depending  on  the  needs  of  the  project: 

•  Description  of  the  assessment  participants 
(including  assessment  team)  and  their 
relationships  (defined  in  2-B); 

•  Description  of  the  budget  and  time  constraints  to 
provide  context  information  for  future  studies; 

•  Human  and  Organizational  Factors  Checklist; 

•  Scenario  details 
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•  Video  and  audio  presentations;  and 

•  Created  models,  spreadsheets,  decision  support 
tools,  etc. 

•  Simple  models  and  tools  (EG  for  the  sponsor  and 
other  parties  to  interactively  explore  interrelationships 
between  the  variables  of  the  study); 

•  Experimentation  Campaign  Plan  (if  the  C2 
Assessment  makes  use  of  a  series  of  linked 
events  such  as  seminars,  wargames,  command 
post  exercises  (CPX),  field  training  exercises 
(FTX)  etc). 

Other  Products 

In  addition  to  study  products  that  are  delivered  to  the 
sponsor  there  are  a  number  of  products  that  best 
practice  demands  are  produced  and  maintained  during 
the  course  of  a  study.  These  include: 

•  A  project  journal; 

•  Study  Management  Plan  (defined  in  4-E) 

•  Data  collection  plan; 

•  Data  analysis  plan;  and 

•  Study  Glossary 

Conclusion 

A  study  is  generally  appraised  based  on  the  quality  of  its 
study  products.  This  Code  of  Best  Practice  aims  to 
highlight  important  areas  that  will  improve  both  the 
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assessment  process,  and  the  quality,  longevity,  and  utility 
of  the  study  products.  The  goal  is  to  make  the  state  of 
practice  one  and  the  same  with  the  state  of  the  art. 

Of  all  the  communities  available  to  us 
there  is  not  one  I  would  want  to  devote 
myself  to,  except  for  the  society  of  the 
true  searches,  which  has  very  few 
living  members  at  any  time . . . 

— Albert  Einstein 
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C2  Command  and  Control 
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C3 

Command,  Control,  and  Communications 
or  Consultation 

C3I 

Command,  Control,  Communications  or 
Consultation,  and  Intelligence 

C4 

Command,  Control,  Communications  or 
Consultation,  and  Computers 

C4I 

Command,  Control,  Communications  or 
Consultation,  Computers,  and  Intelligence 

C4ISR 

Command,  Control,  Communications  or 
Consultation,  Computers,  Intelligence, 
and  Surveillance  and  Reconnaissance 

CIMIC 

Civil-Military  Co-operation 

CIS 

Command  Information  Systems 

CISS 

Center  for  Information  Systems  Security 

COBP 

Code  of  Best  Practice 

CPX 

Command  Post  Exercise 

CTA 

Constructive  Technology  Assessment 

D 

DCEP 

Data  Collection/Engineering  Plan 

DISA 

Defence  Information  Systems  Agency 

DOTMLP 

Doctrine,  Organisation,  Training,  Material, 
Leadership,  Personnel 

DOTMLPF 

Doctrine,  Organisation,  Training,  Material, 
Leadership,  Personnel,  Facilities/Forces 
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F_ 
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FTX 

G_ 

GRR  Generic  Risk  Register 

H _ 

HB(A)  UK  Historical  Branch  (Army) 

HEAT  Headquarters  Effectiveness  Assessment 

System 

HQ  Headquarters 

I _ 

10  International  Organisation 

IRDS  Information  Resource  Dictionary  Systems 
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MAPEX 

MAUT 

MCES 

M-E-M 

METT-TC 

MND(C) 

MoCE 

MoE 

MoFE 
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Multi-Attribute  Utility  Theory 

Modular  Command  and  Control  Evaluation 
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Model-Exercise-Model 

Mission,  Enemy,  Troops,  Terrain,  Troops, 
Time,  and  Civil  considerations 

Multinational  Division  (Centre) 

Measures  of  C2  Effectiveness 

Measures  of  Effectiveness 

Measures  of  Force  Effectiveness 

Measures  of  Merit 


J _ 

JCSC 

JCSE 

JFCOM 


MoM 
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MoP 
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MoPE 
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MORS 

Military  Operations  Research  Society 

M&S 

Modelling  and  Simulation 

M-T-M 

Model-Test-Model 
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NATO 

North  Atlantic  T reaty  Organization 

NC3A 

NATO  C3  (Consultation,  Command  & 
Control)  Agency 

NGO 

Non  Governmental  Organisations 

NL  MOD 

Netherlands  Ministry  of  Defense 

O _ 

OA  Operational  Analysis 

OOTW  Operations  Other  Than  War 

OR  Operations  Research 

p _ 

PESTLE  Political,  Economic,  Social,  Technological, 
Legal,  and  Environmental 

PfP  Partnership  for  Peace 

Policy  Requirements  Land 


PRL 
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Repeatability,  Independence,  Grounding 
in  reality,  Objectivity  of  process, 
Uncertainty,  and  Robustness 

Rules  of  Engagement 

s _ 

SACLANTOA  Supreme  Allied  Command  Atlantic 
Operational  Analysis  Cell 

SCOT  Social  Construction  of  Technology 

SFS  Strike  Force  South 

SFIADE  Shared  Data  Environment 

SFIAPE  Supreme  FIQ  Allied  Powers  Europe 

SMP  Study  Management  Plan 

SOW  Statements  of  Work 

STA  Surveillance  Targeting  and  Acquisition 

STAFFEX  Staff  Exercise 

STS  Science  and  Technology  Studies 

SWOT  Strengths,  Weaknesses,  Opportunities, 

Threats 


R _ 

RIGOUR 

ROE 
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TA 

1 

Technology  Assessment 

TRAC 

TRADOC  Analysis  Center 

TRADOC 

US  Army  T raining  &  Doctrine  Command 

TTP 

Tactics,  Techniques,  and  Procedures 

u 

1 

V 

1 

VTC 

Video  Teleconference 

VV&A 

Verification,  Validation,  and  Accreditation 

VV&C 

Verification,  Validation,  and  Certification 

W 

1 

WBS 

Work  Breakdown  Structure 

X,  Y,  Z 
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The  MORS  Code  of  Ethics  and  Responsibilities  for 
Practitioners  of  Military  Operations  Research 
Military  OR  Professionals  must  aspire  to  be: 

•  Honest,  open  and  trustworthy  in  all  their  relationships; 

•  Reliable  and  consistent  in  the  conduct  of 
assignments  and  responsibilities,  always  doing 
what  is  right  rather  then  expedient; 

•  Objective,  constructive  and  accurate  in  what  they 
say  and  write; 

•  Accountable  for  what  they  do  and  choose  not  to  do; 

•  Respectful  of  the  work  of  others,  giving  due 
credit  and  refraining  from  criticism  of  them 
unless  warranted;  and 

•  Free  from  affiliation  with  others  or  with  activities 
that  would  compromise  them,  their  employers,  or 
the  Society. 
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Human  And 
Organisational 
Issues  Checklist 


This  checklist  presents  a  summary  of  a  particular  set 
of  human  and  organisational  issues  that  is  specifically 
relevant  to  behaviour  and  performance  of  humans  in 
command  and  control  situations.  Its  purpose  is  to 
sensitise  the  analyst  and  help  him  to  assess  whether 
human  or  organisational  issues  are  part  of  the  problem 
domain  and  should  be  addressed  in  the  solution  space. 
Also,  it  should  assist  him  in  identifying  human  sciences 
disciplines  for  consultation. 

It  should  be  pointed  out,  however,  that  this  checklist  is 
preliminary  and  must  not  be  considered  to  be 
comprehensive.  The  field  of  human  sciences  is  too 
large  and  the  set  of  human  and  organisational  issues 
to  ill  defined  to  provide  the  analyst  with  an  exhaustive 
checklist,  at  this  time.  Research  in  this  area  is  ongoing 
and  we  expect  that  later  editions  of  this  document  will 
provide  the  analyst  revised  and  improved  checklists. 

1 .  Human  Issues 

•  Physiological  factors 
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This  refers  to  bio-medical  and  environmental 
factors  that  influence  behaviour  and  performance 

•  Stress 

•  Fatigue  and  lack  of  sleep 

•  Blood  sugar 

•  Fitness  conditions 

•  Weather  conditions 

•  Geographical  terrain  conditions 

•  Ergonomic  factors  (e.g.  performance 
degradation  due  to  working  in  protective  suits) 

Ergonomics  is  a  science  discipline  that  studies 
human  work  and  work  environment  relationships. 

•  Behavioural  factors  (related  to  functioning  in  a 
C2  group) 

•  Social  Competence  &  Experience 

Capability  to  interact  with  others 

•  Communication  skills 

•  Language  skills 

•  Empathy  (Social  Awareness/ 
understanding) 

•  Conflict  handling  style 

•  Frustration  handling  style 

•  Military  Competence  &  Experience 

•  Handling  Danger 
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•  Defining  mission  objectives 

•  Task  Competence  &  Experience 

•  Cognitive  factors 

These  factors  are  related  to  how  humans 
perceive  their  environment,  how  they  give 
meaning  to  what  they  see. 

•  Information  processing  style 

•  Information  processing  capacity 

•  Creating  Situational  Awareness 

•  Individual  decision  making 

•  Risk  tolerance 

•  Pre-disposition! 

•  Receptivity  to  new  information 
(open  /  closed) 

•  Expertise 

•  Prior  training  and  knowledge 
(operational  codes) 

•  Emotional  factors 

•  Morale 

•  Attitude 

•  Separation  from  Family 

•  Fear 

•  Stress 
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•  Resilience  (ability  to  overcome 
negative  feedback) 

•  Leadership  factors 

There  exists  no  clear-cut  concept  of  leadership 
effectiveness,  but  we  do  know  some  of  the 
factors  that  affect  the  effectiveness  of  a  leader 

•  Expectation  and  behaviour  of  superiors 

•  Expectations  and  behaviour  of 
subordinates 

•  Expectations  and  behaviour  of  colleagues 

•  Personality  and  experience 

•  Organisational  Culture  and  policy- with 
regard  to  leadership  and  command. 
Allocation  or  responsibility  and  authority. 

•  Ability  to  Motivate  and  to  Direct  others 

•  Moral  and  Judicial  responsibilities 

•  Coaching  capabilities  (towards 
subordinates) 

2.  Organisational  Issues 

•  Structure 

•  Number  of  echelons  or  layers 

•  Span  of  control  for  nodes 

•  Pattern  of  linkages  between  nodes  (e.g. 
hierarchical,  multi-connected) 

•  Permanent  versus  transitory 
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•  Formal  versus  Informal 

•  Function 

•  Distribution  of  responsibility 

•  Distribution  of  authority 

•  Distribution  of  Information 

•  Functional  Specificity 

•  Ambiguity  in  command  relationships 

•  Capacity 

•  Differences  in  communication  systems  / 
Architectures 

•  Differences  in  information  processing 
systems  /  architectures 

•  Differences  in  field  training  and  operational 
experience 

•  Differences  in  personnel 

•  Experience 

•  Training 

•  Cognitive  ability 

•  Roles 

•  Allocation  of  Responsibility  /  Authority 

•  Role  Conflicts.  The  interference  of  multiple 
Roles  in  one  individual 

•  Sociological  factors 

•  Understanding  of  environment 
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•  Political 

•  Social 

•  Cultural 

•  Interoperability  Issues 

•  National  and  cultural  differences 

•  Organizational  approaches  and  values 

•  Communication  standards  and  technology 

•  Differences  in  perceptions 

•  Organisational  command  style 

•  Decentralised  or  Centralised 

•  Collaborative  versus  Authoritative 

•  Formal  versus  Informal 

•  Command  products:  orders,  objectives, 
missions 

•  Organisational  Culture 

Shapes  patterns  of  organisational  behaviour  and 
reflects  thought  and  activity  patterns  by 
members  of  the  organisation. 

•  Belief  systems 

•  Organisational  norms  &  how  are  they 
expressed 

•  Organisational  values  &  how  are  they 
expressed 

•  Open  to  organisational  learning 
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l.e.  does  the  organisational  culture  embed 
characteristics  that  enable  and  facilitate 
organisational  earning? 

3.  Interaction  between  Human  and 
Organisational  Issues 

•  Group  decision  making 

Groups  can  exert  social  pressure  on  members 
that  affect  the  decision-making  capability  of 
individual  members. 

•  Group  Dynamics 

The  behaviour  of  individuals  in  a  group  will  be 
influenced  by  the  fact  that  they  are  interacting  in 
a  group.  This  interactive  process  contains  a 
number  of  dynamic  dimensions  and  does  effect 
the  result  of  interaction.  A  command  post  is  a 
group  of  people  and  its  performance  is  affected 
by  it  group  dynamics. 

•  Social  interaction  &  communication 

•  Communication 

•  Social  Identity  and  social  conflicts 

•  Individual  dominance/leadership 

•  Cohesion 

•  Teambuilding,  teamwork 

•  Trust  in  group  member’s  competence  and 
loyalty. 


Cultural  factors 
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Culture  is  more  or  less  the  whole  of  beliefs  and 
assumptions,  including  norms  and  values,  about 
things  and  behaviour  in  a  group.  As  such  it 
guides  the  behaviour  of  people,  also  in  the 
military  and  in  command  posts. 

•  Note!  Culture  itself  is  immaterial,  intangible. 
It  can  only  be  distilled  form  observed  actions 
by  members  of  an  organisation  or  societal 
group 

•  Note!  Culture  is  learned  through  interaction 
with  others. 

•  Especially  in  multi-national  OOTW, 
members  of  different  cultural 
background  have  to  interact.  Mutually 
acceptable  norms  of  behaviour  will 
have  to  be  developed  in  the  process. 

•  Socialisation  process 

•  Culture  lag 

Changes  in  the  environment  or  to  the 
organisation  often  require  adaptations  in 
behaviour.  Sometimes  these  changes  are 
required  or  introduced  so  fast  that  an 
organisation  hasn’t  had  the  time  to  the  whole 
of  its  culture  to  adapt  to  the  new  situation, 
i.e,.  to  embed  the  changes  in  its  culture.  This 
may  lead  to  friction. 

•  E.g.1:  The  introduction  of  new 
technologies  may  change  the  way  we 
work.  If  this  is  done  too  fast,  or  without 
thoughtful  guidance  it  may  have  a 
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disruptive  effect  on  behaviour  and  thus 
on  organisational  effectiveness. 

•  E.g.2:  Peace  Keeping  and  War 
Operations  each  require  different 
types  of  behaviours  which  is  often 
quite  different  from  the  type  of 
behaviour  in  the  peaceful  environment 
at  home.  Western  armies  operate  on 
personnel  rotation  schedules  whereby 
individuals  change  frequently  from  one 
situation  to  another.  This  may  also 
result  in  culture  lag  causing  friction. 

•  Sub-culture 

Within  the  larger  set  of  culture,  groups  may 
develop  specialised  cultures,  such  as,  e.g., 
distinct  unit  cultures. 

•  Sometimes  these  sub-cultures  may 
conflict  over  some  issues  with  the 
dominant  organisational  or  societal 
culture.  Understanding  these 
differences  is  often  the  first  step  in 
resolving  these  situations. 

•  Social  Control. 

Individual  (group,  organisational)  members 
watch  and  correct  each  other  if  they  accepted 
procedures  of  behaviour  aren ’t  followed. 

•  If  this  corrective  action  pattern  is  too 
strict,  the  group  becomes  rigid  and  is 
less  able  to  adapt  to  new  situations. 
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•  If  it  is  too  loose,  and  no  culture  or 
norms  are  enforced,  chaos  and 
uncertainty  may  rule  the  group. 

•  Commanders  Intent. 

A  leader  can  effect  the  current  (sub) 
culture  of  his  unit  by  setting  the  example 
and  by  establishing  and  enforcing  required 
procedures  and  behaviour. 

•  Both  must  be  synchronized,  else 
friction  will  occur. 

•  By  action  and  word  a  leader  sets  the 
level  and  direction  of  (social)  control. 

•  Ethics 

•  Cooperability 

•  Effectiveness  of  communication 

•  Willingness  to  co-operate  and  collaborate 

4.  Environmental  factors 

These  factors  affect  individual,  organisational 
and  group  issues. 

•  Noise 

•  Visibility 

•  Temperature  /  humidity 

•  Terrain  type 

•  Infrastructure  in  area  of  operations.  (E.g. 
transport,  communication,  healthcare, 
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agriculture,  food  distribution,  water,  civil 
administration  -infrastructures) 

•  Military  support  infrastructure 

•  Social,  economic,  and  political  situation  in  area 
of  operations 

•  Rules  of  Engagement 

•  Ease  of  interaction 

•  Command  Post  layout 

•  Communication  mechanisms  (e.g.  voice, 
teletype,  VTC) 

•  Geographic  distribution  of  the  command 

Related  Human  Science  Disciplines 

This  list  provides  an  overview  of  the  relevant  scientific 
disciplines  with  the  expertise  to  answer  questions 
arising  in  the  context  of  human  and  organizational 
issues  in  command  and  control.  The  list  implies  a  rough 
taxonomy  of  disciplines  only.  Even  though  boundaries 
between  disciplines  are  frequently  fuzzy,  an  attempt 
was  made  to  minimise  the  overlap. 

1  Psychology 

1.1  Individual  Psychology 

1.1.1  Cognitive  Psychology 

1.1.2  Learning  Psychology 

1.1.3  Action  Theories 
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1 .1 .4  Differential  Psychology 

1.1.5  Psychometrics 

1.2  Social  Psychology 

1.3  Clinical  Psychology 

2  Educational  Sciences  and 
Pedagogics 

3  Sociology 

3.1  Social  Morphology  (e.g.  Social  Stratification, 
Demography) 

3.2  Sociology  of  Political  Power  (See  Also  Political 
Sciences) 

3.3  Mass  Communication 

3.4  Sociological  Methodology  (Empirical  Social 
Research) 

4  Organisational  Sciences 

5  Economic  Sciences 

6  Political  Sciences 

6.1  Science  of  Domestic  Politics 

6.2  Science  of  International  Relations 

6.3  Political  Disaster  Research 
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7  Ergonomics ,  Human  Factors 
Research 

7.1  Physiology  and  Anthropometries 

7.2  Technical  Ergonomics  (Man-Machine-Interface, 
Workspace) 

7.3  Cognitive  Ergonomics  (See  Also  Individual 
Psychology) 

7.4  Crew  Ergonomics  (See  Also  Social  Psychology) 

7.5  Safety  and  Health  Hazard  Prevention 

8  International  Law  of  War, 
International  Humanitarian  Law 

9  Ethics 

9.1  Deontology  (Theories  of  Ethical  Norms  and 
Values) 

9.2  Theories  of  Ethical  Practise  and  Morality 

10  Social  and  Legal  Philosophy 

11  Cultural  Anthropology 
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Alternate 
Definitions  of  Risk 
and  Uncertainty 

A  number  of  different  definitions  of  risk  and 
uncertainty  are  in  common  use,  and  this  can  lead 
to  confusion  and  misunderstanding.  This  COBP  adopts 
specific  working  definitions  of  risk,  uncertainty,  and 
sensitivity;  however,  this  annex  lists  a  range  of  others 
for  information. 

Risk 

Risk  is  defined  in  this  COBP  as  the  possibility  of 
suffering  harm  or  loss.  In  other  words,  exposure  to 
harm  or  loss.  This  includes  an  opportunity  loss. 

Alternate  definitions  of  risk  in  common 
use  include: 

•  not  achieving  your  objective; 

•  the  likelihood  of  not  achieving  your  objective; 

•  a  threat  to  successful  outcome; 
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•  an  assessment  of  the  probability  of  failure; 

•  an  uncertain  future  scenario;  and 

•  a  perception  of  consequential  pain. 

The  alternate  definitions  may  be  seen  as  subtle 
variations  of  each  other  and  are  all  equally  valid. 
However,  this  very  subtlety  can  be  a  source  of 
confusion  and  doubt  within  a  study  unless  a  clear 
working  definition  is  adopted. 

Uncertainty 

Uncertainty  is  defined  in  this  COBP  as  an  inability  to 
determine  a  variable  value  or  system  state  (or  nature) 
or  to  predict  its  future  evolution. 

Alternate  definitions  of  uncertainty  in 
common  use  include: 

•  a  lack  of  clarity  in  the  definition  of  a  system 
or  variable; 

•  a  lack  of  confidence  in  an  assumption  or 
result;  and 

•  a  lack  of  knowledge  about  a  subject  of  interest. 

All  of  these  definitions  have  some  validity,  but  this 
should  not  be  allowed  to  cloud  the  thinking  during 
C2  assessment. 
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Sensitivity 

Sensitivity,  as  used  in  this  COBP,  describes  a  pre¬ 
disposition  to  respond  strongly  to  a  stimulus  or 
variations  in  an  input  factor. 

Alternate  definitions  for  sensitivity  in 
common  use  include: 

•  an  indication  of  the  importance  or  criticality  of  a 
feature  or  variable  of  an  analysis;  and 

•  a  measure  of  the  political  profile  of  the  subject  of 
study. 

Sensitivity,  as  it  relates  to  the  production  of  robust 
analysis,  must  be  clearly  defined  in  terms  of  response 
to  stimulus  rather  than  importance  or  criticality.  In  C2 
systems  many  insensitive  factors  are  critical  and  many 
sensitive  ones  are  relatively  unimportant. 
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